On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome. This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.
Fixes#104372.
* socket_helper.transient_internet() no longer imports nntplib to
catch nntplib.NNTPTemporaryError.
* ssltests.py no longer runs test_nntplib.
* "make quicktest" no longer runs test_nntplib.
* WASM: remove nntplib from OMIT_NETWORKING_FILES.
* Remove mentions to nntplib in the email documentation.
Class editor.IndentSearcher contains all editor references to tokenize module.
Module io tokenize reference cover those other modules.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
In Python 3.8 and prior, `pathlib.Path.__exit__()` marked a path as closed;
some subsequent attempts to perform I/O would raise an IOError. This
functionality was never documented, and had the effect of making `Path`
objects mutable, contrary to PEP 428. In Python 3.9 we made `__exit__()` a
no-op, and in 3.11 `__enter__()` began raising deprecation warnings. Here
we remove both methods.
This matches the GIL releasing behavior of our existing `_hashopenssl`
module, extending it to the HACL* built-ins.
Includes adding comments to better describe the ENTER/LEAVE macros
purpose and explain the lock strategy in both existing and new code.
Apply BOLT optimizations to libpython for shared builds. Most of the C
code is in libpython so it is critical to apply BOLT there fully realize
BOLT benefits.
This change also reworks how BOLT instrumentation is applied. It
effectively removes the readelf based logic added in gh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform BOLT
optimizations without having to manually delete the output binary to
force a new bolt run.
Also:
- add a clean-bolt target for purging BOLT files and hook that up to the
clean target
- .gitignore BOLT related files
Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need to
`rm profile-bolt-stamp` to force a BOLT run because there aren't
appropriate non-phony targets for BOLT's make target to depend on.
To make it easier to iterate on custom BOLT settings, the flags to pass
to instrumentation and application are now defined in configure and can
be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
Fix a race condition in the internal `multiprocessing.process` cleanup
logic that could manifest as an unintended `AttributeError` when calling
`BaseProcess.close()`.
---------
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* Support for conversion specifiers o (octal) and X (uppercase hexadecimal).
* Support for length modifiers j (intmax_t) and t (ptrdiff_t).
* Length modifiers are now applied to all integer conversions.
* Support for wchar_t C strings (%ls and %lV).
* Support for variable width and precision (*).
* Support for flag - (left alignment).
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.
As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
bpo-17258: `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5.
Signed-off-by: Christian Heimes <christian@python.org>
gpshead: I Reworked to be more robust while keeping the idea.
The protocol modification idea remains, but we now take advantage of the
message length as an indicator of legacy vs modern protocol version. No
more regular expression usage. We now default to HMAC-SHA256, but do so
in a way that will be compatible when communicating with older clients
or older servers. No protocol transition period is needed.
More integration tests to verify these claims remain true are required. I'm
unaware of anyone depending on multiprocessing connections between
different Python versions.
---------
Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
This PR updates `math.nextafter` to add a new `steps` argument. The behaviour is as though `math.nextafter` had been called `steps` times in succession.
---------
Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
New additions to the tty library. Functions added: cfmakeraw(), and cfmakecbreak(). The
functions setcbreak() and setraw() now return original termios to save an extra tcgetattr() call.
---------
Signed-off-by: Soumendra Ganguly <soumendraganguly@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
This adds a number of PRIO_DARWIN_* constants to the os module for use with os.setpriority.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Move all of the Python C API calls into the parent process up front
instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from
the post-fork/vfork child process.
Much of this was long overdue. We shouldn't have been using PyTuple and
PyLong APIs within all of these low level functions anyways.
This fixes several ways file descriptors could be leaked from `subprocess.Popen` constructor during error conditions by opening them later and using a context manager "fds to close" registration scheme to ensure they get closed before returning.
---------
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
Fix Zip64 extensions not being properly applied in some cases:
Fixes an issue where adding a small file to a `ZipFile`
object while forcing zip64 extensions causes an extra Zip64 record to be
added to the zip, but doesn't update the `min_version` or file sizes in
the primary central directory header.
Also fixed an edge case in checking if zip64 extensions are required:
This fixes an issue where if data requiring zip64 extensions was added
to an unseekable stream without specifying `force_zip64=True`, zip64
extensions would not be used and a RuntimeError would not be raised when
closing the file (even though the size would be known at that point).
This would result in successfully writing corrupt zip files.
Deciding if zip64 extensions are required outside of the `FileHeader`
function means that both `FileHeader` and `_ZipWriteFile` will always be
in sync. Previously, the `FileHeader` function could enable zip64
extensions without propagating that decision to the `_ZipWriteFile`
class, which would then not correctly write the data descriptor record
or check for errors on close.
If anyone is actually using `ZipInfo.FileHeader` as a public API without
explicitly passing True or False in for zip64, their own code may still be
susceptible to that kind of bug unless they make a similar change to
where the zip64 decision happens.
Fixes#103861
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Add http.client.HTTPConnection method get_proxy_response_headers() - this is a followup to https://github.com/python/cpython/pull/26152 which added it as a non-public attribute. This way we don't pre-compute a headers dictionary that most users will never access. The new method is properly public and documented and triggers full proxy header parsing into a dict only when actually called.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
This implements PEP 695, Type Parameter Syntax. It adds support for:
- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
By convention make targets that don't refer to a file have a dependency
on the fake .PHONY target/file. This ensures that these targets are
always evaluated because there is no rule to create a .PHONY file
and that will force make to think the rule is out of date and needs
to be rebuilt.
This commit consistently associates virtual targets with .PHONY by
declaring the .PHONY dependency immediately above the make rule. This
should avoid race conditions and avoidable rebuilds across multiple make
invocations.
* Pickle the `name` and `args` attributes of AttributeError when present.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Reject HTTP requests with invalid http/x.y version numbers: x or y being non-digits or too-long.
---------
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
These are stubs to be used for adding hypothesis (https://hypothesis.readthedocs.io/en/latest/) tests to the standard library.
When the tests are run in an environment where `hypothesis` and its various dependencies are not installed, the stubs will turn any tests with examples into simple parameterized tests and any tests without examples are skipped.
It also adds hypothesis tests for the `zoneinfo` module, and a Github Actions workflow to run the hypothesis tests as a non-required CI job.
The full hypothesis interface is not stubbed out — missing stubs can be added as necessary.
Co-authored-by: Zac Hatfield-Dodds <zac.hatfield.dodds@gmail.com>
When monitoring LINE events, instrument all instructions that can have a predecessor on a different line.
Then check that the a new line has been hit in the instrumentation code.
This brings the behavior closer to that of 3.11, simplifying implementation and porting of tools.
Use the unused keyword param in the converter to explicitly
mark an argument as unused:
/*[clinic input]
SomeBaseClass.stubmethod
flag: bool(unused=True)
[clinic start generated code]*/
This causes the zipfile module to also consider the character defined by
`os.altsep` (if there is one) to be a path separator and convert it to a
forward slash, as defined by the zip specification.
A logical no-op on all known platforms today as os.altsep is currently only set to a meaningful value on Windows (where it is "/").
`pathlib.Path.glob()` now suppresses all OSError exceptions, except
those raised from calling `is_dir()` on the top-level path.
Previously, `glob()` suppressed ENOENT, ENOTDIR, EBADF and ELOOP
errors and their Windows equivalents. PermissionError was also
suppressed unless it occurred when calling `is_dir()` on the
top-level path. However, the selector would abort prematurely
if a PermissionError was raised, and so `glob()` could return
incomplete results.
This PR removes `_Py_dg_stdnan` and `_Py_dg_infinity` in favour of
using the standard `NAN` and `INFINITY` macros provided by C99.
This change has the side-effect of fixing a bug on MIPS where the
hard-coded value used by `_Py_dg_stdnan` gave a signalling NaN
rather than a quiet NaN.
---------
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
* Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* Fix directory traversal security flaw in uu.decode()
* also check absolute paths and os.altsep
* Add a regression test.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google]
* socket.getnameinfo should release the GIL
* 📜🤖 Added by blurb_it.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)!
Each subinterpreter may now be created with its own GIL (via Py_NewInterpreterFromConfig()). If not so configured then the interpreter will share with the main interpreter--the status quo since subinterpreters were added decades ago. The main interpreter always has its own GIL and subinterpreters from Py_NewInterpreter() will always share with the main interpreter.
Use `io.BufferedWriter` to buffer gzip writes.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Replaces our built-in SHA3 implementation with a verified one from the HACL* project.
This implementation is used when OpenSSL does not provide SHA3 or is not present.
3.11 shiped with a very slow tiny sha3 implementation to get off of the <=3.10 reference implementation that wound up having serious bugs. This brings us back to a reasonably performing built-in implementation consistent with what we've just replaced our other guaranteed available standard hash algorithms with: code from the HACL* project.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Stop de-duplicating results in `_RecursiveWildcardSelector`. A new
`_DoubleRecursiveWildcardSelector` class is introduced which performs
de-duplication, but this is used _only_ for patterns with multiple
non-adjacent `**` segments, such as `path.glob('**/foo/**')`. By avoiding
the use of a set, `PurePath.__hash__()` is not called, and so paths do not
need to be stringified and case-normalised.
Also merge adjacent '**' segments in patterns.
Improve performance of `pathlib.Path.absolute()` and `cwd()` by joining paths only when necessary. Also improve
performance of `PurePath.is_absolute()` on Posix by skipping path parsing and normalization.
`ast.Num`, `ast.Str`, `ast.Bytes`, `ast.Ellipsis` and `ast.NameConstant` now all emit deprecation warnings on import, access, instantation or `isinstance()` checks.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* Remove the unused, private, and undocumented name `_ver` and
the commented-out `print` call.
* Don't add math functions to `__all__`. Beginners should learn
to `import math` to access them.
* Gregor Lindel, who wrote this version of turtle, dropped plans
to implement turtle on another toolkit at least a decade ago.
Drop `_dot` code preparing for this, but add a hint comment.
* `_Screen` is meant to be a singleton class. To enforce that,
it needs either a `__new__` that returns the singleton or
`else...raise` in `__iter__`. Merely removing the `if` clauses
as suggested might break something if a user were to call `_Screen`
directly. Leave the code alone until a problem is evident.
* Turtledemo injects into _Screen both _root and _canvas,
configured as it needs them to be. Making _canvas an `__init__`
option would require skipping some but not all of the lines under
'if _Screen._canvas is None:`. Leave working code alone.
Add `pathlib.PurePath.with_segments()`, which creates a path object from arguments. This method is called whenever a derivative path is created, such as from `pathlib.PurePath.parent`. Subclasses may override this method to share information between path objects.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
This argument allows case-sensitive matching to be enabled on Windows, and
case-insensitive matching to be enabled on Posix.
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
This test can fail unnecessarily. In the test we wait for events on two
file descriptors. This is done in a single call to select.epoll's poll()
function. However, it is valid for the OS to return only one event via
poll() and the next via a subsequent call to poll(). This rarely
happens, but it can cause the test to fail despite properly functioning
polling.
Instead, we poll a second time when necessary.
We now use `_WildcardSelector` to evaluate literal pattern segments, which
allows us to retrieve the real filesystem case.
This change is necessary in order to implement a *case_sensitive* argument
(see GH-81079) and a *follow_symlinks* argument (see GH-77609).
* Remove deprecated classes from pkgutil
* Remove some other PEP 302 obsolescence
* Use find_spec instead of load_module
* Remove more tests of PEP 302 obsolete APIs
* Remove another bunch of tests using obsolete load_modules()
* Remove deleted names from __all__
* Remove obsolete footnote
* imp is removed
* Remove `imp` from generated stdlib names
* What's new and blurb
* Update zipimport documentation for the removed methods
* Fix some Windows tests
* Remove any test (or part of a test) that references `find_module()`.
* Use assertIsNone() / assertIsNotNone() consistently.
* Update Doc/reference/import.rst
* We don't need pkgutil._get_spec() any more either
* test.test_importlib.fixtures.NullFinder
* ...BadLoaderFinder.find_module
* ...test_api.InvalidatingNullFinder.find_module
* ...test.test_zipimport test of z.find_module
* Suppress cross-references to find_loader and find_module
* Suppress cross-references to Finder
* Suppress cross-references to pkgutil.ImpImporter and pkgutil.ImpLoader
---------
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>
The bitwise inversion operator on bool returns the bitwise inversion of the
underlying int value; i.e. `~True == -2` such that `bool(~True) == True`.
It's a common pitfall that users mistake `~` as negation operator and actually
want `not`. Supporting `~` is an artifact of bool inheriting from int. Since there
is no real use-case for the current behavior, let's deprecate `~` on bool and
later raise an error. This removes a potential source errors for users.
Full reasoning: https://github.com/python/cpython/issues/82012#issuecomment-1258705971
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Do not expose the local server's on-disk location from `SimpleHTTPRequestHandler` when generating a directory index. (unnecessary information disclosure)
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
We also expose PyInterpreterConfig. This is part of the PEP 684 (per-interpreter GIL) implementation. We will add docs as soon as we can.
FYI, I'm adding the new config field for per-interpreter GIL in gh-99114.
These segments do not require a `stat()` call, as the selector's
`_select_from()` method is called after we've established that the
parent is a directory.
Check that arguments are strings before calling `os.path.join()`.
Also improve performance of `PurePath(PurePath(...))` while we're in the
area: we now use the *unnormalized* string path of such arguments.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Avoid a potential `ResourceWarning` in `http.client.HTTPConnection`
by closing the proxy / tunnel's CONNECT response explicitly.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
PEP-0682 specified that %-formatting would not support the "z" specifier,
but it was unintentionally allowed for bytes. This PR makes use of the "z"
flag an error for %-formatting in a bytestring.
Issue: #104018
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Add a `teleport` method to `turtle` module turtle instances that acts a lot like `goto`, _but_ ensures the pen is up while warping to the new position to and can control shape filling behavior as part of the jump.
Based on an educator user feature request.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
I changed my name last year, and would like to update my name in the
acknowledgements and git history accordingly.
git-mailmap reference: https://git-scm.com/docs/gitmailmap
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
The default task name is "Task-<counter>" (if no name is passed in during Task creation).
This is initialized in `Task.__init__` (C impl) using string formatting, which can be quite slow.
Actually using the task name in real world code is not very common, so this is wasted init.
Let's defer this string formatting to the first time the name is read (in `get_name` impl),
so we don't need to pay the string formatting cost if the task name is never read.
We don't change the order in which tasks are assigned numbers (if they are) --
the number is set on task creation, as a PyLong instead of a formatted string.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
subprocess's communicate(None) closes stdin of the child process, after
sending no (extra) data. Make asyncio variant do the same.
This fixes issues with processes that waits for EOF on stdin before
continuing.
Using `datetime.datetime.utcnow()` and `datetime.datetime.utcfromtimestamp()` will now raise a `DeprecationWarning`.
We also have removed our internal uses of these functions and documented the change.
This change removes the alias of the 'C' locale to 'en_US'. Because of
this alias, it is currently impossible for an application to use
setlocale() to specify a UTF-8 locale on a system that has no locales
installed, but which supports the C.UTF-8 locale/encoding.
Hugo van Kemenade
Barney Gale
Tomas R
Gregory P. Smith
Jelle Zijlstra
Victor Stinner
Zachary Ware
Steve Dower
Jonathan Protzenko
Pablo Galindo Salgado
Terry Jan Reedy
melanie witt
Nicolas Tessore
Robert O'Shea
Ned Deily
Daniel Fortunov
Sebastian Rittau
Gregory P. Smith
Brandt Bucher
Thomas Wouters
Gregory Szorc
Serhiy Storchaka
Luccccifer
Mark Shannon
gsallam
Marta Gómez Macías
Christian Heimes
Prince Roshan
Matthias Görgens
Christopher Chavez
Soumendra Ganguly
Nikita Sobolev
Youfu Zhang
Ronald Oussoren
Erlend E. Aasland
Carl Meyer
thirumurugan
Guido van Rossum
Alex Waygood
Illia Volochii
cptpcrd
Furkan Onder
Carey Metcalfe
Alexey Namyotkin
Irit Katriel
Charles Machalow
Ben Kallus
Paul Ganssle
James Gerity
Sebastian Berg
JohnJamesUtley
Sam Carroll
Zhang Na
samschott
Nathaniel J. Smith
Eric Snow
Arjun
Itamar Ostricher
chgnrdv
Arthur Pastel
Oleg Iarygin
Tian Gao
Dong-hee Na
Michael Blahay
Shantanu
Gleb Smirnoff
Kevin Krakauer
Petr Viktorin
Adam Turner
Barry Warsaw
Paul Ganssle
Tim Hoffmann
andrei kulakov
Ethan Furman
Juhi Chandalia
Jurica Bradarić
Mariusz Felisiak
Rafael Fontenelle
Thomas Grainger
John Belmonte
Franek Magiera
sunmy2019
Liam Gersten
Pieter Eendebak
Ken Jin
Amethyst Reese
Marek Marczykowski-Górecki
Itayazolay
Reese Hyde
Stefano Rivera
Aidan Melen
Paul Moore
Gordon Messmer