This adds authentication to the forkserver control socket. In the past only filesystem permissions protected this socket from code injection into the forkserver process by limiting access to the same UID, which didn't exist when Linux abstract namespace sockets were used (see issue) meaning that any process in the same system network namespace could inject code. We've since stopped using abstract namespace sockets by default, but protecting our control sockets regardless of type is a good idea.
This reuses the HMAC based shared key auth already used by `multiprocessing.connection` sockets for other purposes.
Doing this is useful so that filesystem permissions are not relied upon and trust isn't implied by default between all processes running as the same UID with access to the unix socket.
### pyperformance benchmarks
No significant changes. Including `concurrent_imap` which exercises `multiprocessing.Pool.imap` in that suite.
### Microbenchmarks
This does _slightly_ slow down forkserver use. How much so appears to depend on the platform. Modern platforms and simple platforms are less impacted. This PR adds additional IPC round trips to the control socket to tell forkserver to spawn a new process. Systems with potentially high latency IPC are naturally impacted more.
Typically a 1-4% slowdown on a very targeted process creation microbenchmark, with a worst case overloaded system slowdown of 20%. No evidence that these slowdowns appear in practical sense. See the PR for details.
Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.
No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
If the cpXXX encoding is not directly implemented in Python, fall back
to use the Windows-specific API codecs.code_page_encode() and
codecs.code_page_decode().
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* Mark almost all reachable objects before doing collection phase
* Add stats for objects marked
* Visit new frames before each increment
* Remove lazy dict tracking
* Update docs
* Clearer calculation of work to do.
Modify the extended attribute tests to write fewer and smaller extended
attributes, in order to fit within filesystems with total xattr limit
of 1 KiB (e.g. ext4 with 1 KiB blocks). Previously, the test would
write over 2 KiB, making it fail with ENOSPC on such systems.
If SRE(match) function terminates abruptly, either because of a signal
or because memory allocation fails, allocated SRE_REPEAT blocks might
be never released.
Co-authored-by: <wjssz@users.noreply.github.com>
The header-folder of the new email API has a long standing known buglet where
if the first token is longer than max_line_length, it puts that token on the next
line. It turns out there is also a *parsing* bug when parsing such a header:
the space prefixing that first, non-empty line gets preserved and tacked on to
the start of the header value, which is not the expected behavior per the RFCs.
The bug arises from the fact that the parser assumed that there would be at
least one token on the line with the header, which is going to be true for
probably every email producer other than the python email library with its
folding buglet. Clearly, though, this is a case that needs to be handled
correctly. The fix is simple: strip the blanks off the start of the whole
value, not just the first physical line of the value.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
This unifies the code for nodejs and the code for the browser. After this
commit, the browser example doesn't work; this will be fixed in a
subsequent update.
* Document that slices can be marshalled
* Deduplicate and organize the list of supported types
in docs
* Organize the type code list in marshal.c, to make
it more obvious that this is a versioned format
* Back-fill some historical info
Co-authored-by: Michael Droettboom <mdboom@gmail.com>
For dlsym(), a return value of NULL does not necessarily indicate
an error [1].
Therefore, to avoid using stale (or NULL) dlerror() values, we must:
1. clear the previous error state by calling dlerror()
2. call dlsym()
3. call dlerror()
If the return value of dlerror() is not NULL, an error occured.
In ctypes we choose to treat a NULL return value from dlsym()
as a "not found" error. This is the same as the fallback
message we use on Windows, Cygwin or when getting/formatting
the error reason fails.
[1]: https://man7.org/linux/man-pages/man3/dlsym.3.html
Signed-off-by: Georgios Alexopoulos <grgalex42@gmail.com>
Signed-off-by: Georgios Alexopoulos <grgalex@ba.uoa.gr>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Fixes a bug where pygettext would attempt
to extract a message from a code like this:
def _(x): pass
This is because pygettext only looks at one
token at a time and '_(x)' looks like a
function call.
However, since 'x' is not a string literal,
it would erroneously issue a warning.
Discard two leading slashes from the beginning of a `file:` URI if they
introduce an empty authority section. As a result, file URIs like
`///etc/hosts` are correctly parsed as `/etc/hosts`.
In strict mode, raise `NotADirectoryError` if we encounter a non-directory
while we still have path parts left to process.
We use a `part_count` variable rather than `len(rest)` because the `rest`
stack also contains markers for unresolved symlinks.
* gh-71936: Fix race condition in multiprocessing.Pool
Proxes of shared objects register a Finalizer in BaseProxy._incref(), and it
will call BaseProxy._decref() when it is GCed. This may cause a race condition
with Pool(maxtasksperchild=None) on Windows.
A connection would be closed and raised TypeError when a GC occurs between
_ConnectionBase._check_writable() and _ConnectionBase._send_bytes() in
_ConnectionBase.send() in the second or later task, and a new object
is allocated that shares the id() of a previously deleted one.
Instead of using the id() of the token (or the proxy), use a unique,
non-reusable number.
Co-Authored-By: Akinori Hattori <hattya@gmail.com>
Hood Chatham
Zachary Ware
Peter Bierma
Mark Shannon
Gregory P. Smith
Brandt Bucher
Jun Komoda
Barney Gale
Hugo van Kemenade
Beomsoo Kim
CoderTCY
Malcolm Smith
sobolevn
Pablo Galindo Salgado
Serhiy Storchaka
Stan U.
Michał Górny
Russell Keith-Magee
Filipe Laíns 🇵🇸
RanKKI
Nadeshiko Manju
Bénédikt Tran
Petr Viktorin
Steve Dower
George Alexopoulos
Sergey B Kirpichev
Victor Stinner
Tomas R.
Marc Culler
CF Bolz-Tereick
Stéphane Bidoul
Seth Michael Larson
Taneli Hukkinen
Ritvik Pasham