bpo-6721: When os.fork() was called while another thread holds a logging lock, the child process may deadlock when it tries to log. This fixes that by acquiring all logging locks before fork and releasing them afterwards.
A regression test that fails before this change is included.
Within the new unittest itself: There is a small _potential_ due to mixing of fork and a thread in the child process if the parent's thread happened to hold a non-reentrant library call lock (malloc?) when the os.fork() happens. buildbots and time will tell if this actually manifests itself in this test or not. :/ A functionality test that avoids that would be a challenge.
An alternate test that isn't trying to produce the deadlock itself but just checking that the release and acquire calls are made would be the next best alternative if so.
[bpo-34658](https://www.bugs.python.org/issue34658): Fix a rare interpreter unhandled exception state SystemError only
seen when using subprocess with a preexec_fn while an after_parent handler has
been registered with os.register_at_fork and the fork system call fails.
https://bugs.python.org/issue34658
Store a weak reference to stream readerfor breaking strong references
It breaks the strong reference loop between reader and protocol and allows to detect and close the socket if the stream is deleted (garbage collected)
When subprocess.Popen() stdin= stdout= or stderr= handles are specified
and appear in pass_fds=, don't close the original fds after dup'ing them.
This implementation and unittest primarily came from @izbyshev (see the PR)
See also b89b52f284
This also removes the old manual p2cread, c2pwrite, and errwrite closing logic
as inheritable flags and _close_open_fds takes care of that properly today without special treatment.
This code is within child_exec() where it is the only thread so there is no
race condition between the dup and _Py_set_inheritable_async_safe call.
Some methods of the SMTP class use mutable default arguments. Specially
`send_message` is affected as it mutates one of the args by appending items
to it, which has side effects on further calls.
Release GIL on grp.getgrnam(), grp.getgrgid(), pwd.getpwnam() and
pwd.getpwuid() if reentrant variants of these functions are available.
Patch by William Grzybowski.
Fail `test_semaphore_tracker_sigint` if no warnings are expected and one is received.
Fix race condition when the child receives SIGINT before it can register signal handlers for it.
The race condition occurs when the parent calls
`_semaphore_tracker.ensure_running()` (which in turn spawns the
semaphore_tracker using `_posixsubprocess.fork_exec`), the child
registers the signal handlers and the parent tries to kill the child.
What seem to happen is that in some slow systems, the parent sends the
signal to kill the child before the child protects against the signal.
Make mixed-type `%` and `//` operations involving `Fraction` and `float` objects behave like all other mixed-type arithmetic operations: first the `Fraction` object is converted to a `float`, then the `float` operation is performed as normal. This fixes some surprising corner cases, like `Fraction('1/3') % inf` giving a NaN.
Thanks Elias Zamaria for the patch.
The current C implementations **crash** if the input includes a surrogate
Unicode code point, which is not possible to encode in UTF-8.
Important notes:
1. It is possible to pass a non-UTF-8 string as a separator to the
`.isoformat()` methods.
2. The pure-Python `datetime.fromisoformat()` implementation accepts
strings with a surrogate as the separator.
In `datetime.fromisoformat()`, in the special case of non-UTF-8 separators,
this implementation will take a performance hit by making a copy of the
input string and replacing the separator with 'T'.
Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
Co-authored-by: Paul Ganssle <paul@ganssle.io>
Introduce a configure check for strsignal(3) which defines HAVE_STRSIGNAL for
signalmodule.c. Add some common signals on HP-UX. This change applies for
Windows and HP-UX.
os.readlink() now accepts path-like and bytes objects on Windows.
Previously, support for path-like and bytes objects was only
implemented on Unix.
This commit also merges Unix and Windows implementations of
os.readlink() in one function and adds basic unit tests to increase
test coverage of the function.
ZipFile can zip files older than 1980-01-01 and newer than 2107-12-31 using
a new strict_timestamps parameter at the cost of setting the timestamp
to the limit.
* Fix integer overflow in os.readv(), os.writev(), os.preadv()
and os.pwritev() and in os.sendfile() with headers or trailers
arguments (on BSD-based OSes and MacOS).
* Fix sending the part of the file in os.sendfile() on MacOS.
Using the trailers argument could cause sending more bytes from
the input file than was specified.
Thanks Ned Deily for testing on 32-bit MacOS.
* help(hashlib) didn't work because of incorrect module name in blake2b and
blake2s classes.
* Constructors blake2*(), sha3_*(), shake_*() and keccak_*() incorrectly
accepted keyword argument "string" for binary data, but documented as
accepting the "data" keyword argument. Now this parameter is positional-only.
* Keyword-only parameters in blake2b() and blake2s() were not documented as
keyword-only.
* Default value for some parameters of blake2b() and blake2s() was None,
which is not acceptable value.
* The length argument for shake_*.digest() was wrapped out to 32 bits.
* The argument for shake_128.digest() and shake_128.hexdigest() was not
positional-only as intended.
* TypeError messages for incorrect arguments in all constructors sha3_*(),
shake_*() and keccak_*() incorrectly referred to sha3_224.
Also made the following enhancements:
* More accurately specified input and result types for strings, bytes and
bytes-like objects.
* Unified positional parameter names for update() and constructors.
* Improved formatting.
Various asyncio internals expect that the default executor is a
`ThreadPoolExecutor`, so deprecate passing anything else to
`loop.set_default_executor()`.
* Always return bytes from _HackedGetData.get_data().
Ensure the imp.load_source shim always returns bytes by reopening the file in
binary mode if needed. Hash-based pycs have to receive the source code in bytes.
It's tempting to change imp.get_suffixes() to always return 'rb' as a mode, but
that breaks some stdlib tests and likely 3rdparty code, too.
Most of the change involves fixing up the test suite, which previously made
the assumption that there wouldn't be a new line if the input didn't end in
one.
Contributed by Ammar Askar.
tarfile._Stream has two buffer for compressed and uncompressed data.
Those buffers are not aligned so unnecessary bytes slicing happens
for every reading chunks.
This commit bypass compressed buffering.
In this benchmark [1], user time become 250ms from 300ms.
[1]: https://bugs.python.org/msg320763
subprocess.Popen now copies the startupinfo argument to leave it
unchanged: it will modify the copy, so that the same STARTUPINFO
object can be used multiple times.
Add subprocess.STARTUPINFO.copy() method.
During buffered read, use a list followed by join instead of extending a bytes object.
This is how it was done before but changed in commit b506dc32c1.
The Opera Browser was using a outdated command line invocation that resulted in an incorrect URL being opened in the browser when requested using the webbrowser module.
* Correct the arguments passed to the Opera Browser when opening a new URL.
* `flags` is indeed deprecated, but there is a validation on its value for
backwards compatibility reasons. This adds mention of this in the docs.
* The docs say that `sizehint` is deprecated and ignored, but it is still
used when `epoll_create1()` is unavailable. This adds mention of this in
the docs.
* `sizehint=-1` is acceptable again, and is replaced with `FD_SETSIZE-1`.
This is needed to have a default value available at the Python level,
since `FD_SETSIZE` is not exposed to Python. (see: bpo-31938)
* Reject `sizehint=0` since it is invalid to pass on to `epoll_create()`.
The relevant tests have also been updated.
Fix a race condition in Popen of
multiprocessing.popen_spawn_win32. The child process now duplicates
the read end of pipe instead of "stealing" it.
Previously, the read end of pipe was "stolen" by the child process,
but it leaked a handle if the child process had been terminated
before it could steal the handle from the parent process.
bpo-33671
* use memoryview() with size == file size on Windows, see https://github.com/python/cpython/pull/7160#discussion_r195405230
* release intermediate (sliced) memoryview immediately
* replace "OSX" occurrences with "macOS"
* add some unittests for copyfileobj()
with debuglevel=1 only the header keys got printed. With
this change the header values get printed as well and the single
header entries get '\n' as a separator.
Hangul composition check boundaries are wrong for the second character
([0x1161, 0x1176) instead of [0x1161, 0x1176]) and third character ((0x11A7, 0x11C3)
instead of [0x11A7, 0x11C3]).
* have shutil.copyfileobj use sendfile() if possible
* refactoring: use ctx manager
* add test with non-regular file obj
* emulate case where file size can't be determined
* reference _copyfileobj_sendfile directly
* add test for offset() at certain position
* add test for empty file
* add test for non regular file dst
* small refactoring
* leave copyfileobj() alone in order to not introduce any incompatibility
* minor refactoring
* remove old test
* update docstring
* update docstring; rename exception class
* detect platforms which only support file to socket zero copy
* don't run test on platforms where file-to-file zero copy is not supported
* use tempfiles
* reset verbosity
* add test for smaller chunks
* add big file size test
* add comment
* update doc
* update whatsnew doc
* update doc
* catch Exception
* remove unused import
* add test case for error on second sendfile() call
* turn docstring into comment
* add one more test
* update comment
* add Misc/NEWS entry
* get rid of COPY_BUFSIZE; it belongs to another PR
* update doc
* expose posix._fcopyfile() for OSX
* merge from linux branch
* merge from linux branch
* expose fcopyfile
* arg clinic for the win implementation
* convert path type to path_t
* expose CopyFileW
* fix windows tests
* release GIL
* minor refactoring
* update doc
* update comment
* update docstrings
* rename functions
* rename test classes
* update doc
* update doc
* update docstrings and comments
* avoid do import nt|posix modules if unnecessary
* set nt|posix modules to None if not available
* micro speedup
* update description
* add doc note
* use better wording in doc
* rename function using 'fastcopy' prefix instead of 'zerocopy'
* use :ref: in rst doc
* change wording in doc
* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies
* move CopyFileW in _winapi and actually expose CopyFileExW instead
* fix line endings
* add tests for mode bits
* add docstring
* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies
* update doc to reflect new changes
* update doc
* adjust tests on win
* fix argument clinic error
* update doc
* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python
* osx / copyfile: use path_t instead of char
* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)
* add same file test
* add test for same file
* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted
* turn PermissionError into appropriate SameFileError
* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError
* honour follow_symlinks arg when using CopyFileEx
* update Misc/NEWS
* expose CreateDirectoryEx mock
* change C type
* CreateDirectoryExW actual implementation
* provide specific makedirs() implementation for win
* fix typo
* skeleton for SetNamedSecurityInfo
* get security info for src path
* finally set security attrs
* add unit tests
* mimick os.makedirs() behavior and raise if dst dir exists
* set 2 paths for OSError object
* set 2 paths for OSError object
* expand windows test
* in case of exception on os.sendfile() set filename and filename2 exception attributes
* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX
* update doc
* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation
* use bytearray() and readinto()
* use memoryview() with bytearray()
* refactoring + introduce a new _fastcopy_binfileobj() fun
* remove CopyFileEx and other C wrappers
* remove code related to CopyFileEx
* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible
* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable
* use ctx manager for memoryview()
* update doc
* remove outdated doc
* remove last CopyFileEx remnants
* OSX - use fcopyfile(3) instead of copyfile(3)
...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.
* update doc
When attempting to base64-decode a payload of invalid length (1 mod 4),
properly recognize and handle it. The given data will be returned as-is,
i.e. not decoded, along with a new defect, InvalidBase64LengthDefect.
Before Python 3.6, os.path.abspath(None) used to report an AttributeError which was properly caught inside site.abs_paths, making it ignore __main__, one of sys.modules, which has __file__ and __cached__ set to None. With 3.6, os.path.abspath(None) raises TypeError instead which site.abs_path was not expecting. This resulted in an uncaught exception if a user had PYTHONSTARTUP set and the application called site.main() which a number of third-party programs do.
A datetime object d is aware if d.tzinfo is not None and
d.tzinfo.utcoffset(d) does not return None. If d.tzinfo is None,
or if d.tzinfo is not None but d.tzinfo.utcoffset(d) returns None,
d is naive.
This commit ensures that instances with non-None d.tzinfo, but
d.tzinfo.utcoffset(d) returning None are treated as naive.
In addition, C acceleration code will raise TypeError if
d.tzinfo.utcoffset(d) returns an object with the type other than
timedelta.
* Updated the documentation.
Assume that the term "naive" is defined elsewhere and remove the
not entirely correct clarification. Thanks, Tim.
- bugfix and test for fragile metavar handling in argparse (see
bpo-24089, bpo-14046, bpo-25058, bpo-11874)
- also fixes some incorrect tests that did not make 1-element tuples correctly
The cancellation of an overlapped WSARecv() has a race condition
which causes data loss because of the current implementation of
proactor in asyncio.
No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport
to work around the race condition.
Remove the optimized recv_into() implementation to get simple
implementation of pause_reading() using the single _pending_data
attribute.
Move _feed_data_to_bufferred_proto() to protocols.py.
Remove set_protocol() method which became useless.
With 3.7+, dictionary are ordered by design. Configparser still uses
collections.OrderedDict, which is unnecessary. This updates the module
to use the standard dict implementation by default, and changes the
docs and tests to match.
* Fix AttributeError (not all SSL exceptions have 'errno' attribute)
* Increase default handshake timeout from 10 to 60 seconds
* Make sure start_tls can be cancelled correctly
* Make sure any error in SSLProtocol gets propagated (instead of just being logged)
Future.set_result and Future.set_exception now raise InvalidStateError
if the futures are not pending or running. This mirrors the behavior
of asyncio.Future, and prevents AssertionErrors in asyncio.wrap_future
when set_result is called multiple times.
Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately. This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time. This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:
condition = asyncio.Condition()
async with condition:
await asyncio.wait_for(condition.wait(), timeout=0.5)
Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.
To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way. This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
Fix a race condition in SSLProtocol.connection_made() of
asyncio.sslproto: start immediately the handshake instead of using
call_soon(). Previously, data_received() could be called before the
handshake started, causing the handshake to hang or fail.
In this commit:
* Support BufferedProtocol in set_protocol() and start_tls()
* Fix proactor to cancel readers reliably
* Update tests to be compatible with OpenSSL 1.1.1
* Clarify BufferedProtocol docs
* Bump TLS tests timeouts to 60 seconds; eliminate possible race from start_serving
* Rewrite test_start_tls_server_1
bpo-26510 in 3.7.0a2 changed the behavior of argparse to make
subparsers required by default, returning to the behavior of 2.7
and 3.2. The behavior was changed in 3.3 to be no longer required.
While it might make more sense to have the default to required,
compatibility with 3.3 through 3.6 is probably less disruptive
than trying to reintroduce compatibility with 2.7 at this point.
This change restores the 3.6 behavior.
TLS 1.3 behaves slightly different than TLS 1.2. Session tickets and TLS
client cert auth are now handled after the initialy handshake. Tests now
either send/recv data to trigger session and client certs. Or tests
ignore ConnectionResetError / BrokenPipeError on the server side to
handle clients that force-close the socket fd.
To test TLS 1.3, OpenSSL 1.1.1-pre7-dev (git master + OpenSSL PR
https://github.com/openssl/openssl/pull/6340) is required.
Signed-off-by: Christian Heimes <christian@python.org>
Change TLS 1.3 cipher suite settings for compatibility with OpenSSL
1.1.1-pre6 and newer. OpenSSL 1.1.1 will have TLS 1.3 cipers enabled by
default.
Also update multissltests and Travis config to test with latest OpenSSL.
Signed-off-by: Christian Heimes <christian@python.org>
* bpo-33263 Fix FD leak in _SelectorSocketTransport. (GH-6450)
Under particular circumstances _SelectorSocketTransport can try to add a reader
even the transport is already being closed. This can lead to FD leak and
invalid stated of the following connections. Fixed the SelectorSocketTransport
to add the reader only if the trasport is still active.
uuid._ipconfig_getnode did not validate the maximum length of the value,
so long as the value had the same type of formatting as a MAC address.
This let it select DUIDs as MAC addresses. It now requires an exact
length match.
The proactor event loop has a race condition when reading with
pausing/resuming. `resume_reading()` unconditionally schedules the read
function to read from the current future. If `resume_reading()` was
called before the previously scheduled done callback fires, this results
in two attempts to get the data from the most recent read and an
assertion failure. This commit tracks whether or not `resume_reading`
needs to reschedule the callback to restart the loop, preventing a
second attempt to read the data.
The ssl module now contains OP_NO_RENEGOTIATION constant, available with
OpenSSL 1.1.0h or 1.1.1.
Note, OpenSSL 1.1.0h hasn't been released yet.
Signed-off-by: Christian Heimes <christian@python.org>
The urllib.robotparser's __str__ representation now includes wildcard
entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra
newlines that were being appended to the end of the string.
* bpo-33251: ConfigParser.items no longer returns items present in vars.
Documentation for `ConfigParser.items()` states:
'Items present in vars no longer appear in the result.'
This fix aligns behaviour to that specified in the documentation.
Starting with glibc 2.27.9000-xxx, sigaddset() can return EINVAL for some
reserved signal numbers between 1 and NSIG. The `range(1, NSIG)` idiom
is commonly used to select all signals for blocking with `pthread_sigmask`.
So we ignore the sigaddset() return value until we expose sigfillset()
to provide a better idiom.
* Now uses pickle protocol 4
* Doesn't wrap the grammar's `__dict__` in ordered dictionaries anymore as
dictionaries in Python 3.6+ are ordered by default
This still produces deterministic pickles (that hash the same with MD5).
Tested with different PYTHONHASHSEED values.
The pydoc CLI assumed -m pydoc would add the empty string
to sys.path, and hence got confused when it switched to
adding the full initial working directory instead.
This refactors the pydoc CLI path manipulation to be
more testable, and ensures it won't accidentally
remove the standard library directory containing
pydoc itself from sys.path.
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
* bpo-29613: Added support for SameSite cookies
Implemented as per draft
https://tools.ietf.org/html/draft-west-first-party-cookies-07
* Documented SameSite
And suggestions by members.
* Missing space :(
* Updated News and contributors
* Added version changed details.
* Fix in documentation
* fix in documentation
* Clubbed test cases for same attribute into single.
* Updates
* Style nits + expand tests
* review feedback
This makes performance better and produces shorter pickles. This change is backwards compatible up to the oldest currently supported version of Python (3.4).
This also fixespython/typing#512
This also fixespython/typing#511
As was discussed in both issues, some typing forms deserve to be treated
as immutable by copy and pickle modules, so that:
* copy(X) is X
* deepcopy(X) is X
* loads(dumps(X)) is X # pickled by reference
This PR adds such behaviour to:
* Type variables
* Special forms like Union, Any, ClassVar
* Unsubscripted generic aliases to containers like List, Mapping, Iterable
This not only resolves inconsistencies mentioned in the issues, but also
improves backwards compatibility with previous versions of Python
(including 3.6).
Note that this requires some dances with __module__ for type variables
(similar to NamedTuple) because the class TypeVar itself is define in typing,
while type variables should get module where they were defined.
https://bugs.python.org/issue32873
bpo-32844: subprocess: Fix a potential misredirection of a low fd to stderr.
When redirecting, subprocess attempts to achieve the following state:
each fd to be redirected to is less than or equal to the fd
it is redirected from, which is necessary because redirection
occurs in the ascending order of destination descriptors.
It fails to do so in a couple of corner cases,
for example, if 1 is redirected to 2 and 0 is closed in the parent.
LibreSSL 2.7 introduced OpenSSL 1.1.0 API. The ssl module now detects
LibreSSL 2.7 and only provides API shims for OpenSSL < 1.1.0 and
LibreSSL < 2.7.
Documentation updates and fixes for failing tests will be provided in
another patch set.
Signed-off-by: Christian Heimes <christian@python.org>
If a non-dataclass derives from a frozen dataclass, allow attributes to be set.
Require either all of the dataclasses in a class hierarchy to be frozen, or all non-frozen.
Store `@dataclass` parameters on the class object under `__dataclass_params__`. This is needed to detect frozen base classes.
New tests also added.
I also made the comments in line with the builtin Grammar/Grammar. PEP 306 was
withdrawn, Kees Blom's railroad program has been lost to the sands of time for
at least 16 years now (I found a python-dev post from people looking for it).
fstat may block for long time if the file descriptor is on a
non-responsive NFS server, hanging all threads. Most fstat() calls are
handled by _Py_fstat(), releasing the GIL internally, but but
_Py_fstat_noraise() does not release the GIL, and most calls release the
GIL explicitly around it.
This patch fixes last 2 calls to _Py_fstat_no_raise(), avoiding hangs
when calling:
- mmap.mmap()
- os.urandom()
- random.seed()
In some conditions the standard streams will be None or closed in the child process (for example if using "pythonw" instead of "python" on Windows). Avoid failing with a non-0 exit code in those conditions.
Report and initial patch by poxthegreat.
OpenSSL 1.1 has introduced a new API to set the minimum and maximum
supported protocol version. The API is easier to use than the old
OP_NO_TLS1 option flags, too.
Since OpenSSL has no call to set minimum version to highest supported,
the implementation emulate maximum_version = MINIMUM_SUPPORTED and
minimum_version = MAXIMUM_SUPPORTED by figuring out the minumum and
maximum supported version at compile time.
Signed-off-by: Christian Heimes <christian@python.org>
Direct instantiation of SSLSocket and SSLObject objects is now prohibited.
The constructors were never documented, tested, or designed as public
constructors. The SSLSocket constructor had limitations. For example it was
not possible to enabled hostname verification except was
ssl_version=PROTOCOL_TLS_CLIENT with cert_reqs=CERT_REQUIRED.
SSLContext.wrap_socket() and SSLContext.wrap_bio are the recommended API
to construct SSLSocket and SSLObject instances. ssl.wrap_socket() is
also deprecated.
The only test case for direct instantiation was added a couple of days
ago for IDNA testing.
Signed-off-by: Christian Heimes <christian@python.org>
* bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes
Misc fixes and workarounds for compatibility with OpenSSL 1.1.1-pre1 and
TLS 1.3 support. With OpenSSL 1.1.1, Python negotiates TLS 1.3 by
default. Some test cases only apply to TLS 1.2. Other tests currently
fail because the threaded or async test servers stop after failure.
I'm going to address these issues when OpenSSL 1.1.1 reaches beta.
OpenSSL 1.1.1 has added a new option OP_ENABLE_MIDDLEBOX_COMPAT for TLS
1.3. The feature is enabled by default for maximum compatibility with
broken middle boxes. Users should be able to disable the hack and CPython's test suite needs
it to verify default options.
Signed-off-by: Christian Heimes <christian@python.org>
unsafe_hash=False is now the default. It is the same behavior as the old hash=None parameter. unsafe_hash=True will try to add __hash__. If it already exists, TypeError is raised.
The ctypes module used to depend on indirect linking for dlopen. The shared
extension is now explicitly linked against libdl on platforms with dl.
Signed-off-by: Christian Heimes <christian@python.org>
The ssl module now detects missing NPN support in LibreSSL.
Co-Authored-By: Bernard Spil <brnrd@FreeBSD.org>
Signed-off-by: Christian Heimes <christian@python.org>
* The SSLSocket is no longer implemented on top of SSLObject to
avoid an extra level of indirection.
* Owner and session are now handled in the internal constructor.
* _ssl._SSLSocket now uses the same method names as SSLSocket and
SSLObject.
* Channel binding type check is now handled in C code. Channel binding
is always available.
The patch also changes the signature of SSLObject.__init__(). In my
opinion it's fine. A SSLObject is not a user-constructable object.
SSLContext.wrap_bio() is the only valid factory.
ssl.match_hostname() has been simplified and no longer depends on re and
ipaddress module for wildcard and IP addresses. Error reporting for invalid
wildcards has been improved.
Signed-off-by: Christian Heimes <christian@python.org>
os.dup2() tests for dup3() system call availability at runtime,
but doesn't remember the result across calls, repeating
the test on each call with inheritable=False.
Since the caller of os.dup2() is expected to hold the GIL,
fix this by making the variable holding the test result static.
fnmatch.translate() no longer produces patterns which contain set
operations.
Sets starting with '[' or containing '--', '&&', '~~' or '||' will
be interpreted differently in regular expressions in future versions.
Currently they emit warnings. fnmatch.translate() now avoids producing
patterns containing such sets by accident.
Fix a rare but potential pre-exec child process deadlock in subprocess on POSIX systems when marking file descriptors inheritable on exec in the child process. This bug appears to have been introduced in 3.4 with the inheritable file descriptors support.
This also changes Python/fileutils.c `set_inheritable` to use the "slow" two `fcntl` syscall path instead of the "fast" single `ioctl` syscall path when asked to be async signal safe (by way of being asked not to raise exceptions). `ioctl` is not a POSIX async-signal-safe approved function.
ref: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
This allows the compression level to be specified when writing zipfiles
(for the entire file *and* overridden on a per-file basis).
Contributed by Bo Bayles
Do not allow receiving a SIGINT to cause the subprocess module to trigger an
immediate SIGKILL of the child process. SIGINT is normally sent to all child
processes by the OS at the same time already as was the established normal
behavior in 2.7 and 3.2. This behavior change was introduced during the fix to https://bugs.python.org/issue12494 and is generally surprising to command line
tool users who expect other tools launched in child processes to get their own
SIGINT and do their own cleanup.
In Python 3.3-3.6 subprocess.call and subprocess.run would immediately
SIGKILL the child process upon receiving a SIGINT (which raises a
KeyboardInterrupt). We now give the child a small amount of time to
exit gracefully before resorting to a SIGKILL.
This is also the case for subprocess.Popen.__exit__ which would
previously block indefinitely waiting for the child to die. This was
hidden from many users by virtue of subprocess.call and subprocess.run
sending the signal immediately.
Behavior change: subprocess.Popen.__exit__ will not block indefinitely
when the exiting exception is a KeyboardInterrupt. This is done for
user friendliness as people expect their ^C to actually happen. This
could cause occasional orphaned Popen objects when not using `call` or
`run` with a child process that hasn't exited.
Refactoring involved: The Popen.wait method deals with the
KeyboardInterrupt second chance, existing platform specific internals
have been renamed to _wait().
Also fixes comment typos.
Fix socket(fileno=fd) by auto-detecting the socket's family, type,
and proto from the file descriptor. The auto-detection can be overruled
by passing in family, type, and proto explicitly.
Without the fix, all socket except for TCP/IP over IPv4 are basically broken:
>>> s = socket.create_connection(('www.python.org', 443))
>>> s
<socket.socket fd=3, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('2003:58:bc4a:3b00:56ee:75ff:fe47:ca7b', 59730, 0, 0), raddr=('2a04:4e42:1b::223', 443, 0, 0)>
>>> socket.socket(fileno=s.fileno())
<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('2003:58:bc4a:3b00::%2550471192', 59730, 0, 2550471192), raddr=('2a04:4e42:1b:0:700c:e70b:ff7f:0%2550471192', 443, 0, 2550471192)>
Signed-off-by: Christian Heimes <christian@python.org>
Until now Python used a hard coded white list of default TLS cipher
suites. The old approach has multiple downsides. OpenSSL's default
selection was completely overruled. Python did neither benefit from new
cipher suites (ChaCha20, TLS 1.3 suites) nor blacklisted cipher suites.
For example we used to re-enable 3DES.
Python now defaults to OpenSSL DEFAULT cipher suite selection and black
lists all unwanted ciphers. Downstream vendors can override the default
cipher list with --with-ssl-default-suites.
Signed-off-by: Christian Heimes <christian@python.org>
Class authors no longer need to specify repr=False if they want to provide a custom __repr__ for dataclasses. The same thing applies for the other dunder methods that the dataclass decorator adds. If dataclass finds that a dunder methods is defined in the class, it will not overwrite it.
bpo-31399: Let OpenSSL verify hostname and IP
The ssl module now uses OpenSSL's X509_VERIFY_PARAM_set1_host() and
X509_VERIFY_PARAM_set1_ip() API to verify hostname and IP addresses.
* Remove match_hostname calls
* Check for libssl with set1_host, libssl must provide X509_VERIFY_PARAM_set1_host()
* Add documentation for OpenSSL 1.0.2 requirement
* Don't support OpenSSL special mode with a leading dot, e.g. ".example.org" matches "www.example.org". It's not standard conform.
* Add hostname_checks_common_name
Signed-off-by: Christian Heimes <christian@python.org>
Benjamin Peterson
Eric V. Smith
Gregory P. Smith
Andrew Svetlov
Tony Flury
Ethan Furman
orlnub123
Sergey Fedoseev
Oren Milman
Zackery Spytz
Zachary Ware
Rahul Jha
Matthias Bussonnier
Berker Peksag
Julien Malard
Pablo Aguiar
William Grzybowski
Tal Einat
Alexander Buchkovsky
Victor Stinner
Pablo Galindo
Michael Felt
Naris R
Carl Meyer
Elias Zamaria
Serhiy Storchaka
Vladimir Matveev
Paul Ganssle
Michael Osipov
Alexey Izbyshev
Vadim Pushtaev
Alex Grönholm
Anthony Sottile
Marcel Plch
MartinAltmayer
Raymond Hettinger
twisteroid ambassador
Elvis Pranskevichus
Mickaël Schoentgen
Franz Wöllert
Dong-hee Na
Matěj Cepl
Jason R. Coombs
Ammar Askar
INADA Naoki
hajoscher
Bumsik Kim
Xtreak
Terry Jan Reedy
Yury Selivanov
Ned Deily
Giampaolo Rodola
Marco Strigl
ValeriyaSinevich
Wonsup Yoon
Cheryl Sabella
Steve Weber
Alexander Belopolsky
Thomas Kluyver
wim glenn
Scott Sanderson
arikrupnik
John Reese
Vinay Sajip
jhaydaman
Géry Ogam
jimmylai
Ethan Smith
Christian Heimes
sth
Vlad Starostin
Aaron Hall, MBA
Mario Corchero
CtrlZvi
Ivan Levkivskyi
Steve Dower
Nina Zakharenko
Petr Viktorin
Christopher Beacham
Segev Finer
Michael Lazar
sblondon
Andrés Delfino
Antoine Pitrou
Michael Selik
Brett Cannon
Chris Bradbury
Paul Moore
Aaron Ang
Wolfgang Maier
Łukasz Langa
Nick Coghlan
jdemeyer
Mark Nemec
Alex Gaynor
Julien Palard
Alexey Izbyshev
Garvit Khatri
Thomas Moreau
Xiang Zhang
Matt Eaton
aetracht
Antoine Pietri
Nir Soffer
Emanuele Gaifas
Joffrey F
animalize
Éric Araujo
Kyle Altendorf
Bar Harel
Коренберг Марк
Alan D Moore
Bernhard M. Wiedemann
Stéphane Wirtel
John Jolly
Anders Lorentsen
Bo Bayles
jayyyin
Nitish Chandra
jcea
Mads Jensen