When subprocess.Popen() stdin= stdout= or stderr= handles are specified
and appear in pass_fds=, don't close the original fds after dup'ing them.
This implementation and unittest primarily came from @izbyshev (see the PR)
See also b89b52f284
This also removes the old manual p2cread, c2pwrite, and errwrite closing logic
as inheritable flags and _close_open_fds takes care of that properly today without special treatment.
This code is within child_exec() where it is the only thread so there is no
race condition between the dup and _Py_set_inheritable_async_safe call.
(cherry picked from commit ce34410b8b)
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google]
Many type object initializations labeled a field "tp_size" in the
comment, but the name of that field is tp_basicsize..
(cherry picked from commit 0e0bc4e221)
Co-authored-by: Peter Eisentraut <peter@eisentraut.org>
Pass the user/group name as Unicode to the formatting function,
instead of always decoding a bytes string from UTF-8..
(cherry picked from commit 28658485a5)
Co-authored-by: William Grzybowski <wg@FreeBSD.org>
Fix for invalid assert on big output of multiprocessing.Process.
(cherry picked from commit 266f4904a2)
Co-authored-by: Alexander Buchkovsky <olex.buchkovsky@gmail.com>
* _Py_InitializeCore() now sets the LC_CTYPE locale to the user
preferred locale before checking if the C locale should be coerced
or not in _PyCoreConfig_Read().
* Fix pymain_read_conf(): remember if the C locale has been coerced
when the configuration should be read again if the encoding has
changed.
(cherry picked from commit 2c8ddcf4f1)
* The UTF-8 Mode is now also enabled by the "POSIX" locale, not only
by the "C" locale.
* On FreeBSD, Py_DecodeLocale() and Py_EncodeLocale() now also forces
the ASCII encoding if the LC_CTYPE locale is "POSIX", not only if
the LC_CTYPE locale is "C".
* test_utf8_mode.test_cmd_line() checks also that the command line
arguments are decoded from UTF-8 when the the UTF-8 Mode is enabled
with POSIX locale or C locale.
(cherry picked from commit 5cb258950c)
Avoids an integer underflow in the time module's year handling code.
(cherry picked from commit 76be0fffff)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
The current C implementations **crash** if the input includes a surrogate
Unicode code point, which is not possible to encode in UTF-8.
Important notes:
1. It is possible to pass a non-UTF-8 string as a separator to the
`.isoformat()` methods.
2. The pure-Python `datetime.fromisoformat()` implementation accepts
strings with a surrogate as the separator.
In `datetime.fromisoformat()`, in the special case of non-UTF-8 separators,
this implementation will take a performance hit by making a copy of the
input string and replacing the separator with 'T'.
Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
Co-authored-by: Paul Ganssle <paul@ganssle.io>
(cherry picked from commit 096329f0b2)
Co-authored-by: Paul Ganssle <pganssle@users.noreply.github.com>
bpo-34170, bpo-34207: pymain_read_conf() now sets Py_UTF8Mode to
config->utf8_mode. pymain_read_conf() calls indirectly
Py_DecodeLocale() and Py_EncodeLocale() which depend on Py_UTF8Mode.
(cherry picked from commit 89487f51b8)
The missing NULL check was reported by Svace static analyzer.
(cherry picked from commit cdbf50cba1)
Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
* -X dev: it is now possible to override the memory allocator using
PYTHONMALLOC even if the developer mode is enabled.
* Add _Py_InitializeFromConfig()
* Add _Py_Initialize_ReadEnvVars() to set global configuration
variables from environment variables
* Fix the code to initialize Python: Py_Initialize() now also reads
environment variables
* _Py_InitializeCore() can now be called twice: the second call
only replaces the configuration.
* Write unit tests on Py_Initialize() and the different ways to
configure Python
* The isolated mode now always sets Py_IgnoreEnvironmentFlag and
Py_NoUserSiteDirectory to 1.
* pymain_read_conf() now saves/restores the configuration
if the encoding changed
* Fix integer overflow in os.readv(), os.writev(), os.preadv()
and os.pwritev() and in os.sendfile() with headers or trailers
arguments (on BSD-based OSes and MacOS).
* Fix sending the part of the file in os.sendfile() on MacOS.
Using the trailers argument could cause sending more bytes from
the input file than was specified.
Thanks Ned Deily for testing on 32-bit MacOS.
(cherry picked from commit 9d5727326a)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* help(hashlib) didn't work because of incorrect module name in blake2b and
blake2s classes.
* Constructors blake2*(), sha3_*(), shake_*() and keccak_*() incorrectly
accepted keyword argument "string" for binary data, but documented as
accepting the "data" keyword argument. Now this parameter is positional-only.
* Keyword-only parameters in blake2b() and blake2s() were not documented as
keyword-only.
* Default value for some parameters of blake2b() and blake2s() was None,
which is not acceptable value.
* The length argument for shake_*.digest() was wrapped out to 32 bits.
* The argument for shake_128.digest() and shake_128.hexdigest() was not
positional-only as intended.
* TypeError messages for incorrect arguments in all constructors sha3_*(),
shake_*() and keccak_*() incorrectly referred to sha3_224.
Also made the following enhancements:
* More accurately specified input and result types for strings, bytes and
bytes-like objects.
* Unified positional parameter names for update() and constructors.
* Improved formatting.
(cherry picked from commit f1d36d8efa)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
On Windows, passing a negative value to local results in an OSError because localtime_s on Windows does not support negative timestamps. Unfortunately this means that fold detection for timestamps between 0 and max_fold_seconds will result in this OSError since we subtract max_fold_seconds from the timestamp to detect a fold. However, since we know there haven't been any folds in the interval [0, max_fold_seconds) in any timezone, we can hackily just forego fold detection for this time range on Windows.
(cherry picked from commit 96d1e69a12)
Co-authored-by: Ammar Askar <ammar_askar@hotmail.com>
Py_Main() can again be called after Py_Initialize(), as in Python
3.6. The new configuration is ignored, except of
_PyMainInterpreterConfig.argv which is used to update sys.argv.
(cherry picked from commit fb47bca9ee)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
They can be exposed when some C API calls fail due to lack of
memory.
* Failed Py_BuildValue() could cause an assertion error in the
following TextIOWrapper.tell().
* input_chunk could be decrefed twice in TextIOWrapper.seek()
after failed Py_BuildValue().
* initvalue could leak in StringIO.__getstate__() after failed
PyDict_Copy().
(cherry picked from commit fdb5a50ef3)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* `flags` is indeed deprecated, but there is a validation on its value for
backwards compatibility reasons. This adds mention of this in the docs.
* The docs say that `sizehint` is deprecated and ignored, but it is still
used when `epoll_create1()` is unavailable. This adds mention of this in
the docs.
* `sizehint=-1` is acceptable again, and is replaced with `FD_SETSIZE-1`.
This is needed to have a default value available at the Python level,
since `FD_SETSIZE` is not exposed to Python. (see: bpo-31938)
* Reject `sizehint=0` since it is invalid to pass on to `epoll_create()`.
The relevant tests have also been updated.
(cherry picked from commit 0cdf5f4289)
Co-authored-by: Tal Einat <taleinat+github@gmail.com>
bz2, lzma: When Decompressor.__init__() is called twice, free the old
lock to not leak memory.
(cherry picked from commit 9b7cf75721)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Fix "LC_ALL=C python3.7 -V": reset properly the command line parser
when the encoding changes after reading the Python configuration.
Fix pymain_read_conf(): use memset(0) to reset properly cmdline.
(cherry picked from commit 6c5a4b3156)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Hangul composition check boundaries are wrong for the second character
([0x1161, 0x1176) instead of [0x1161, 0x1176]) and third character ((0x11A7, 0x11C3)
instead of [0x11A7, 0x11C3]).
(cherry picked from commit d134809cd3)
Co-authored-by: Wonsup Yoon <pusnow@me.com>
The documentation for CERT_NONE, CERT_OPTIONAL, and CERT_REQUIRED were
misleading and partly wrong. It fails to explain that OpenSSL behaves
differently in client and server mode. Also OpenSSL does validate the
cert chain everytime. With SSL_VERIFY_NONE a validation error is not
fatal in client mode and does not request a client cert in server mode.
Also discourage people from using CERT_OPTIONAL in client mode.
(cherry picked from commit ef24b6c54d)
Co-authored-by: Christian Heimes <christian@python.org>
A datetime object d is aware if d.tzinfo is not None and
d.tzinfo.utcoffset(d) does not return None. If d.tzinfo is None,
or if d.tzinfo is not None but d.tzinfo.utcoffset(d) returns None,
d is naive.
This commit ensures that instances with non-None d.tzinfo, but
d.tzinfo.utcoffset(d) returning None are treated as naive.
In addition, C acceleration code will raise TypeError if
d.tzinfo.utcoffset(d) returns an object with the type other than
timedelta.
* Updated the documentation.
Assume that the term "naive" is defined elsewhere and remove the
not entirely correct clarification. Thanks, Tim.
(cherry picked from commit 877b23202b)
Co-authored-by: Alexander Belopolsky <abalkin@users.noreply.github.com>
Move the floor() call into fbound() to call floor() on a double
rather than an int. The change should enhance the rounding.
Document also (int)double rounding mode.
(cherry picked from commit 45e4efba7f)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Fix a crash in Python initialization when parsing the command line
options.
Fix memcpy() size parameter: previously, we read one wchar_t after
the end of _PyOS_optarg. Moreover, don't copy the trailingg NUL
character: we write it manually anyway.
Thanks Christoph Gohlke for the bug report and the fix!
(cherry picked from commit 58d1683255)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Use uuid_enc_be() if available to encode UUID to bytes as big endian.
(cherry picked from commit 17d8830312)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* Fixed a leak when the GC fails to add an object with __del__ into
the gc.garbage list.
* PyGC_Collect() can now be called when an exception is set and
preserves it.
* Fixed an undefined behavior with comparing a dead pointer with NULL.
(cherry picked from commit 301e3cc8a5)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Fix the following bugs in the C implementation:
* get_future_loop() silenced all exceptions raised when look up the get_loop
attribute, not just an AttributeError.
* enter_task() silenced all exceptions raised when look up the current task,
not just a KeyError.
* repr() was called for a borrowed link in enter_task() and task_step_impl().
* str() was used instead of repr() in formatting one error message (in
Python implementation too).
* There where few reference leaks in error cases.
(cherry picked from commit 6655354afc)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
The editline emulation needs to be initialized *after* the name is
defined. This fixes the long open issue.
(cherry picked from commit c2f082e9d1)
Co-authored-by: Zvezdan Petkovic <zpetkovic@acm.org>
The ssl module now contains OP_NO_RENEGOTIATION constant, available with
OpenSSL 1.1.0h or 1.1.1.
Note, OpenSSL 1.1.0h hasn't been released yet.
Signed-off-by: Christian Heimes <christian@python.org>
(cherry picked from commit 67c4801663)
Co-authored-by: Christian Heimes <christian@python.org>
Starting with glibc 2.27.9000-xxx, sigaddset() can return EINVAL for some
reserved signal numbers between 1 and NSIG. The `range(1, NSIG)` idiom
is commonly used to select all signals for blocking with `pthread_sigmask`.
So we ignore the sigaddset() return value until we expose sigfillset()
to provide a better idiom.
(cherry picked from commit 25038ecfb6)
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Setup modules are no longer built with -DPy_BUILD_CORE by default,
as using that flag may now require including additional internal-only header files.
Instead, only the modules that specifically need it use that setting.
(cherry picked from commit 063db62aab)
Co-authored-by: xdegaye <xdegaye@gmail.com>
bpo-32844: subprocess: Fix a potential misredirection of a low fd to stderr.
When redirecting, subprocess attempts to achieve the following state:
each fd to be redirected to is less than or equal to the fd
it is redirected from, which is necessary because redirection
occurs in the ascending order of destination descriptors.
It fails to do so in a couple of corner cases,
for example, if 1 is redirected to 2 and 0 is closed in the parent.
(cherry picked from commit 0e7144b064)
Co-authored-by: Alexey Izbyshev <izbyshev@users.noreply.github.com>
Harden ssl module against LibreSSL CVE-2018-8970.
X509_VERIFY_PARAM_set1_host() is called with an explicit namelen. A new test
ensures that NULL bytes are not allowed.
Signed-off-by: Christian Heimes <christian@python.org>
(cherry picked from commit d02ac25ab0)
Co-authored-by: Christian Heimes <christian@python.org>
LibreSSL 2.7 introduced OpenSSL 1.1.0 API. The ssl module now detects
LibreSSL 2.7 and only provides API shims for OpenSSL < 1.1.0 and
LibreSSL < 2.7.
Documentation updates and fixes for failing tests will be provided in
another patch set.
Signed-off-by: Christian Heimes <christian@python.org>
(cherry picked from commit 4ca0739c9d)
Co-authored-by: Christian Heimes <christian@python.org>
bpo-33018 (GH-5944) fixed bpo-32999 too. So fc7df0e6 is not required
anymore. Revert it except test case.
(cherry picked from commit f757b72b25)
Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
This improves error message for situations when a non-class is
checked w.r.t. an abstract base class.
(cherry picked from commit 40472dd42d)
Co-authored-by: jab <jab@users.noreply.github.com>
fstat may block for long time if the file descriptor is on a
non-responsive NFS server, hanging all threads. Most fstat() calls are
handled by _Py_fstat(), releasing the GIL internally, but but
_Py_fstat_noraise() does not release the GIL, and most calls release the
GIL explicitly around it.
This patch fixes last 2 calls to _Py_fstat_no_raise(), avoiding hangs
when calling:
- mmap.mmap()
- os.urandom()
- random.seed()
(cherry picked from commit 4484f9dca9)
Co-authored-by: Nir Soffer <nirsof@gmail.com>
Multi-phase initialized modules allow m_traverse to be called while the
module is still being initialized, so module authors may need to account
for that.
(cherry picked from commit c2b0b12d1a)
Co-authored-by: Marcel Plch <gmarcel.plch@gmail.com>
OpenSSL 1.1 has introduced a new API to set the minimum and maximum
supported protocol version. The API is easier to use than the old
OP_NO_TLS1 option flags, too.
Since OpenSSL has no call to set minimum version to highest supported,
the implementation emulate maximum_version = MINIMUM_SUPPORTED and
minimum_version = MAXIMUM_SUPPORTED by figuring out the minumum and
maximum supported version at compile time.
Signed-off-by: Christian Heimes <christian@python.org>
(cherry picked from commit 698dde16f6)
Co-authored-by: Christian Heimes <christian@python.org>
* bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes
Misc fixes and workarounds for compatibility with OpenSSL 1.1.1-pre1 and
TLS 1.3 support. With OpenSSL 1.1.1, Python negotiates TLS 1.3 by
default. Some test cases only apply to TLS 1.2. Other tests currently
fail because the threaded or async test servers stop after failure.
I'm going to address these issues when OpenSSL 1.1.1 reaches beta.
OpenSSL 1.1.1 has added a new option OP_ENABLE_MIDDLEBOX_COMPAT for TLS
1.3. The feature is enabled by default for maximum compatibility with
broken middle boxes. Users should be able to disable the hack and CPython's test suite needs
it to verify default options.
Signed-off-by: Christian Heimes <christian@python.org>
(cherry picked from commit 05d9fe32a1)
Co-authored-by: Christian Heimes <christian@python.org>
Miss Islington (bot)
Benjamin Peterson
William Grzybowski
Victor Stinner
Alexey Izbyshev
Serhiy Storchaka
Steve Dower
Eric Snow
Pablo Galindo