* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe
When used to run a new executable image, fork() is not a good choice
for process creation, especially if the parent has a large working set:
fork() needs to copy page tables, which is slow, and may fail on systems
where overcommit is disabled, despite that the child is not going to
touch most of its address space.
Currently, subprocess is capable of using posix_spawn() instead, which
normally provides much better performance. However, posix_spawn() does not
support many of child setup operations exposed by subprocess.Popen().
Most notably, it's not possible to express `close_fds=True`, which
happens to be the default, via posix_spawn(). As a result, most users
can't benefit from faster process creation, at least not without
changing their code.
However, Linux provides vfork() system call, which creates a new process
without copying the address space of the parent, and which is actually
used by C libraries to efficiently implement posix_spawn(). Due to sharing
of the address space and even the stack with the parent, extreme care
is required to use vfork(). At least the following restrictions must hold:
* No signal handlers must execute in the child process. Otherwise, they
might clobber memory shared with the parent, potentially confusing it.
* Any library function called after vfork() in the child must be
async-signal-safe (as for fork()), but it must also not interact with any
library state in a way that might break due to address space sharing
and/or lack of any preparations performed by libraries on normal fork().
POSIX.1 permits to call only execve() and _exit(), and later revisions
remove vfork() specification entirely. In practice, however, almost all
operations needed by subprocess.Popen() can be safely implemented on
Linux.
* Due to sharing of the stack with the parent, the child must be careful
not to clobber local variables that are alive across vfork() call.
Compilers are normally aware of this and take extra care with vfork()
(and setjmp(), which has a similar problem).
* In case the parent is privileged, special attention must be paid to vfork()
use, because sharing an address space across different privilege domains
is insecure[1].
This patch adds support for using vfork() instead of fork() on Linux
when it's possible to do safely given the above. In particular:
* vfork() is not used if credential switch is requested. The reverse case
(simple subprocess.Popen() but another application thread switches
credentials concurrently) is not possible for pure-Python apps because
subprocess.Popen() and functions like os.setuid() are mutually excluded
via GIL. We might also consider to add a way to opt-out of vfork() (and
posix_spawn() on platforms where it might be implemented via vfork()) in
a future PR.
* vfork() is not used if `preexec_fn != None`.
With this change, subprocess will still use posix_spawn() if possible, but
will fallback to vfork() on Linux in most cases, and, failing that,
to fork().
[1] https://ewontfix.com/7
Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from v1 to v2.2.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/upload-artifact/releases">actions/upload-artifact's releases</a>.</em></p>
<blockquote>
<h2>v2.2.0</h2>
<ul>
<li>Support for artifact retention</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="27bce4eee7"><code>27bce4e</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/112">#112</a> from thboop/main</li>
<li><a href="f8b42f7ab4"><code>f8b42f7</code></a> update licensed files</li>
<li><a href="2106e8cf10"><code>2106e8c</code></a> update contributing.md</li>
<li><a href="db66798ebc"><code>db66798</code></a> Ignore Generated Files in Git PR's</li>
<li><a href="d359fd0772"><code>d359fd0</code></a> Manual Verification of licenses</li>
<li><a href="350822c32f"><code>350822c</code></a> Add Licensed Workflow and config</li>
<li><a href="abecf4abf4"><code>abecf4a</code></a> Updated README.md (<a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/118">#118</a>)</li>
<li><a href="604e071d21"><code>604e071</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/126">#126</a> from yacaovsnc/main</li>
<li><a href="4560c23b39"><code>4560c23</code></a> Check for invalid retention-days input</li>
<li><a href="59018c2f85"><code>59018c2</code></a> Add an option to specify retention period</li>
<li>Additional commits viewable in <a href="https://github.com/actions/upload-artifact/compare/v1...27bce4eee761b5bc643f46a8dfb41b430c8d05f6">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Automerge-Triggered-By: GH:Mariatta
Bumps [actions/cache](https://github.com/actions/cache) from v1 to v2.1.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/cache/releases">actions/cache's releases</a>.</em></p>
<blockquote>
<h2>v2.1.2</h2>
<ul>
<li>Adds input to limit the chunk upload size, useful for self-hosted runners with slower upload speeds</li>
<li>No-op when executing on GHES</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="d1255ad936"><code>d1255ad</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/cache/issues/424">#424</a> from actions/dhadka/upload-chunk-size</li>
<li><a href="68cfb2ccb7"><code>68cfb2c</code></a> Add units to description</li>
<li><a href="cce3c03a74"><code>cce3c03</code></a> Add new input to action.yml</li>
<li><a href="4bceb75b5b"><code>4bceb75</code></a> Use parseInt instead of Number to handle empty strings</li>
<li><a href="a6f1f4b32e"><code>a6f1f4b</code></a> Adds input for upload chunk size</li>
<li><a href="d606e039ae"><code>d606e03</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/cache/issues/421">#421</a> from actions/dhadka/ghes</li>
<li><a href="d3e4f218f3"><code>d3e4f21</code></a> Use warning instead of info</li>
<li><a href="55a5894438"><code>55a5894</code></a> Update dist</li>
<li><a href="3f6dfcbcc4"><code>3f6dfcb</code></a> Merge branch 'main' of <a href="http://github.com/actions/cache">http://github.com/actions/cache</a> into dhadka/ghes</li>
<li><a href="0f71d4ac9a"><code>0f71d4a</code></a> Add tests for isGhes</li>
<li>Additional commits viewable in <a href="https://github.com/actions/cache/compare/v1...d1255ad9362389eac595a9ae406b8e8cb3331f16">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Automerge-Triggered-By: GH:Mariatta
The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.) It take ~40 seconds on my machine to analyze the full CPython code base.
Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.
https://bugs.python.org/issue36876
Add documentation to help diagnose CDLL dependent DLL loading errors
on windows for OSError with message:
"[WinError 126] The specified module could not be found"
This error is otherwise difficult to diagnose.
@ericsnowcurrently This PR will change the following:
In the library documentation importlib.rst:
- `module.__package__` can be `module.__name__` for packages;
- `spec.parent` can be `spec.__name__` for packages;
- `spec.loader` is not `None` for namespaces packages.
In the language documentation import.rst:
- `spec.loader` is not `None` for namespace packages.
Automerge-Triggered-By: GH:warsaw
This is a trivial fix to [bpo-39416](), which didn't come up until it was already committed
```
Change "Numeric" to "numeric".
I believe this is trivial enough to not need an issue or a NEWS entry, although
I'm unclear on what branches the original pull request received backports.
```
Automerge-Triggered-By: GH:merwok
[bpo-39416](): Document string representations of the Numeric classes
This is a change to the specification of the Python language.
The idea here is to put sane minimal limits on the Python language's default
representations of its Numeric classes. That way "Marty's Robotic Massage Parlor
and Python Interpreter" implementation of Python won't do anything too
crazy.
Some discussion in the email thread:
Subject: Documenting Python's float.__str__()
https://mail.python.org/archives/list/python-dev@python.org/thread/FV22TKT3S2Q3P7PNN6MCXI6IX3HRRNAL/
* Add _newline_ parameter to `pathlib.Path.write_text()`
* Update documentation of `pathlib.Path.write_text()`
* Add test case for `pathlib.Path.write_text()` calls with _newline_ parameter passed
Automerge-Triggered-By: GH:methane
Clarify in the subprocess documentation how searching for the executable to run works, noting that ``sys.executable`` is the recommended way to find the current interpreter.
Andre Delfino
Mariatta
Terry Jan Reedy
Raymond Hettinger
Brett Cannon
Alexey Izbyshev
Jacob Neil Taylor
dependabot[bot]
John Losito
Mark Shannon
Christopher Marchfelder
Eric Snow
Zackery Spytz
Cheryl Sabella
Lysandros Nikolaou
Petr Viktorin
Batuhan Taskaya
Philippe Ombredanne
larryhastings
Christian Heimes
Dong-hee Na
Vladimir Matveev
Pablo Galindo
Géry Ogam
kpinc
Raúl Cumplido
Paul Ganssle
Максим
Gregory P. Smith
kj
Florian Dahlitz
Paul Moore
Saiyang Gou
Serhiy Storchaka