* bpo-33263 Fix FD leak in _SelectorSocketTransport. (GH-6450)
Under particular circumstances _SelectorSocketTransport can try to add a reader
even the transport is already being closed. This can lead to FD leak and
invalid stated of the following connections. Fixed the SelectorSocketTransport
to add the reader only if the trasport is still active.
uuid._ipconfig_getnode did not validate the maximum length of the value,
so long as the value had the same type of formatting as a MAC address.
This let it select DUIDs as MAC addresses. It now requires an exact
length match.
The proactor event loop has a race condition when reading with
pausing/resuming. `resume_reading()` unconditionally schedules the read
function to read from the current future. If `resume_reading()` was
called before the previously scheduled done callback fires, this results
in two attempts to get the data from the most recent read and an
assertion failure. This commit tracks whether or not `resume_reading`
needs to reschedule the callback to restart the loop, preventing a
second attempt to read the data.
The ssl module now contains OP_NO_RENEGOTIATION constant, available with
OpenSSL 1.1.0h or 1.1.1.
Note, OpenSSL 1.1.0h hasn't been released yet.
Signed-off-by: Christian Heimes <christian@python.org>
Instead of passing configure args such as --without-gcc or --with-icc,
instead prefer to rely on the native way of finding the compiler:
passing CC (or CPP or CXX depending).
This allows configure to find the correct compiler instead of having to
be explicitly told. It also more correctly builds on both macOS and
FreeBSD since the system compiler is used by default (cc)
The urllib.robotparser's __str__ representation now includes wildcard
entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra
newlines that were being appended to the end of the string.
The hash implementation casts the input pointer to uint64_t* and directly reads
from this, which may cause unaligned accesses. Use memcpy() instead so this code
will not crash with SIGBUS on sparc.
https://bugs.gentoo.org/show_bug.cgi?id=636400
Remove the paragraph where we explain that os.utime() does not support a
directory as path under Windows. Patch by Jan-Philip Gehrcke
Co-authored-by: Jan-Philip Gehrcke <jgehrcke@gmail.com>
In text and entry boxes, this affects selection by double-click,
movement left/right by control-left/right, and deletion left/right
by control-BACKSPACE/DEL.
METH_NOARGS functions need only a single argument but they are cast
into a PyCFunction, which takes two arguments. This triggers an
invalid function cast warning in gcc8 due to the argument mismatch.
Fix this by adding a dummy unused argument.
External importers were being added in both phases of the import
system initialisation.
They're only supposed to be added in the second phase, after the
import machinery has been appropriately configured.
* bpo-33251: ConfigParser.items no longer returns items present in vars.
Documentation for `ConfigParser.items()` states:
'Items present in vars no longer appear in the result.'
This fix aligns behaviour to that specified in the documentation.
Starting with glibc 2.27.9000-xxx, sigaddset() can return EINVAL for some
reserved signal numbers between 1 and NSIG. The `range(1, NSIG)` idiom
is commonly used to select all signals for blocking with `pthread_sigmask`.
So we ignore the sigaddset() return value until we expose sigfillset()
to provide a better idiom.
Setup modules are no longer built with -DPy_BUILD_CORE by default,
as using that flag may now require including additional internal-only header files.
Instead, only the modules that specifically need it use that setting.
Fix clang ubsan (undefined behavior sanitizer) warnings in dictobject.c by
adjusting how the internal struct _dictkeysobject shared keys structure is
declared.
This remains ABI compatible. We get rid of the union at the end of the
struct being used for conveinence to avoid typecasting in favor of char[]
variable length array at the end of a struct. This is known to clang to be
used for variable sized objects and will not cause an undefined behavior
problem. Similarly, char arrays do not have strict aliasing undefined
behavior when cast.
PEP-007 does not currently list variable length arrays (VLAs) as allowed
in our subset of C99. If this turns out to be a problem, the fix to this is
to change the char `dk_indices[]` into `dk_indices[1]` and restore the
three size computation subtractions this change removes:
`- Py_MEMBER_SIZE(PyDictKeysObject, dk_indices)`
If this works as is I'll make a separate PR to update PEP-007.
* Now uses pickle protocol 4
* Doesn't wrap the grammar's `__dict__` in ordered dictionaries anymore as
dictionaries in Python 3.6+ are ordered by default
This still produces deterministic pickles (that hash the same with MD5).
Tested with different PYTHONHASHSEED values.
The pydoc CLI assumed -m pydoc would add the empty string
to sys.path, and hence got confused when it switched to
adding the full initial working directory instead.
This refactors the pydoc CLI path manipulation to be
more testable, and ensures it won't accidentally
remove the standard library directory containing
pydoc itself from sys.path.
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
* bpo-29613: Added support for SameSite cookies
Implemented as per draft
https://tools.ietf.org/html/draft-west-first-party-cookies-07
* Documented SameSite
And suggestions by members.
* Missing space :(
* Updated News and contributors
* Added version changed details.
* Fix in documentation
* fix in documentation
* Clubbed test cases for same attribute into single.
* Updates
* Style nits + expand tests
* review feedback
* bpo-33201: Modernize "Extension types" doc
* Split tutorial and other topics
* Some small fixes
* Address some review comments
* Rename noddy* to custom* and shoddy to sublist
* Fix markup
This makes performance better and produces shorter pickles. This change is backwards compatible up to the oldest currently supported version of Python (3.4).
Build and link with private copy of Tcl/Tk 8.6 for the macOS 10.6+
installer. The 10.9+ installer variant already does this. This means that
the Python 3.7 provided by the python.org macOS installers no longer need or
use any external versions of Tcl/Tk, either system-provided or user-
installed, such as ActiveTcl.
test_asyncio hangs indefinitely on macOS 10.13.2+ on `read_pty_output()`
using the KqueueSelector. Closing `proto.transport` (as is done in
`write_pty_output()`) seems to fix it.
(cherry picked from commit 12f74d8608)
Co-authored-by: Nathan Henrie <n8henrie@users.noreply.github.com>
Also, re-enable test_read_pty_output on macOS.
This also fixespython/typing#512
This also fixespython/typing#511
As was discussed in both issues, some typing forms deserve to be treated
as immutable by copy and pickle modules, so that:
* copy(X) is X
* deepcopy(X) is X
* loads(dumps(X)) is X # pickled by reference
This PR adds such behaviour to:
* Type variables
* Special forms like Union, Any, ClassVar
* Unsubscripted generic aliases to containers like List, Mapping, Iterable
This not only resolves inconsistencies mentioned in the issues, but also
improves backwards compatibility with previous versions of Python
(including 3.6).
Note that this requires some dances with __module__ for type variables
(similar to NamedTuple) because the class TypeVar itself is define in typing,
while type variables should get module where they were defined.
https://bugs.python.org/issue32873
bpo-32844: subprocess: Fix a potential misredirection of a low fd to stderr.
When redirecting, subprocess attempts to achieve the following state:
each fd to be redirected to is less than or equal to the fd
it is redirected from, which is necessary because redirection
occurs in the ascending order of destination descriptors.
It fails to do so in a couple of corner cases,
for example, if 1 is redirected to 2 and 0 is closed in the parent.
Historically, -m added the empty string as sys.path
zero, meaning it resolved imports against the current
working directory, the same way -c and the interactive
prompt do.
This changes the sys.path initialisation to add the
*starting* working directory as sys.path[0] instead,
such that changes to the working directory while the
program is running will have no effect on imports
when using the -m switch.
- new test case for pre-initialization of sys.warnoptions and sys._xoptions
- restored ability to call these APIs prior to Py_Initialize
- updated the docs for the affected APIs to make it clear they can be
called before Py_Initialize
- also enhanced the existing embedding test cases
to check for expected settings in the sys module
Harden ssl module against LibreSSL CVE-2018-8970.
X509_VERIFY_PARAM_set1_host() is called with an explicit namelen. A new test
ensures that NULL bytes are not allowed.
Signed-off-by: Christian Heimes <christian@python.org>
LibreSSL 2.7 introduced OpenSSL 1.1.0 API. The ssl module now detects
LibreSSL 2.7 and only provides API shims for OpenSSL < 1.1.0 and
LibreSSL < 2.7.
Documentation updates and fixes for failing tests will be provided in
another patch set.
Signed-off-by: Christian Heimes <christian@python.org>
* Added new opcode END_ASYNC_FOR.
* Setting global StopAsyncIteration no longer breaks "async for" loops.
* Jumping into an "async for" loop is now disabled.
* Jumping out of an "async for" loop no longer corrupts the stack.
* Simplify the compiler.
If a non-dataclass derives from a frozen dataclass, allow attributes to be set.
Require either all of the dataclasses in a class hierarchy to be frozen, or all non-frozen.
Store `@dataclass` parameters on the class object under `__dataclass_params__`. This is needed to detect frozen base classes.
Multi-phase initialized modules allow m_traverse to be called while the
module is still being initialized, so module authors may need to account
for that.
Creating backup files with ~ suffix can be undesirable in some environment,
such as when building RPM packages. Instead of requiring the user to remove
those files manually, option -n was added, that simply disables this feature.
-n was selected because 2to3 has the same option with this behavior.
New tests also added.
I also made the comments in line with the builtin Grammar/Grammar. PEP 306 was
withdrawn, Kees Blom's railroad program has been lost to the sands of time for
at least 16 years now (I found a python-dev post from people looking for it).
fstat may block for long time if the file descriptor is on a
non-responsive NFS server, hanging all threads. Most fstat() calls are
handled by _Py_fstat(), releasing the GIL internally, but but
_Py_fstat_noraise() does not release the GIL, and most calls release the
GIL explicitly around it.
This patch fixes last 2 calls to _Py_fstat_no_raise(), avoiding hangs
when calling:
- mmap.mmap()
- os.urandom()
- random.seed()
In some conditions the standard streams will be None or closed in the child process (for example if using "pythonw" instead of "python" on Windows). Avoid failing with a non-0 exit code in those conditions.
Report and initial patch by poxthegreat.
Fix a crash on fork when using a custom memory allocator (ex: using
PYTHONMALLOC env var).
_PyGILState_Reinit() and _PyInterpreterState_Enable() now use the
default RAW memory allocator to allocate a new interpreters mutex on
fork.
Like Python, IDLE optionally runs one startup file in the Shell window
before presenting the first interactive input prompt. For IDLE,
option -s runs a file named in environmental variable IDLESTARTUP or
PYTHONSTARTUP; -r file runs file. Python sets __file__ to the startup
file name before running the file and unsets it before the first
prompt. IDLE now does the same when run normally, without the -n
option.
* Prevent low-grade poplib REDOS (CVE-2018-1060)
The regex to test a mail server's timestamp is susceptible to
catastrophic backtracking on long evil responses from the server.
Happily, the maximum length of malicious inputs is 2K thanks
to a limit introduced in the fix for CVE-2013-1752.
A 2KB evil response from the mail server would result in small slowdowns
(milliseconds vs. microseconds) accumulated over many apop calls.
This is a potential DOS vector via accumulated slowdowns.
Replace it with a similar non-vulnerable regex.
The new regex is RFC compliant.
The old regex was non-compliant in edge cases.
* Prevent difflib REDOS (CVE-2018-1061)
The default regex for IS_LINE_JUNK is susceptible to
catastrophic backtracking.
This is a potential DOS vector.
Replace it with an equivalent non-vulnerable regex.
Also introduce unit and REDOS tests for difflib.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
OpenSSL 1.1 has introduced a new API to set the minimum and maximum
supported protocol version. The API is easier to use than the old
OP_NO_TLS1 option flags, too.
Since OpenSSL has no call to set minimum version to highest supported,
the implementation emulate maximum_version = MINIMUM_SUPPORTED and
minimum_version = MAXIMUM_SUPPORTED by figuring out the minumum and
maximum supported version at compile time.
Signed-off-by: Christian Heimes <christian@python.org>
Direct instantiation of SSLSocket and SSLObject objects is now prohibited.
The constructors were never documented, tested, or designed as public
constructors. The SSLSocket constructor had limitations. For example it was
not possible to enabled hostname verification except was
ssl_version=PROTOCOL_TLS_CLIENT with cert_reqs=CERT_REQUIRED.
SSLContext.wrap_socket() and SSLContext.wrap_bio are the recommended API
to construct SSLSocket and SSLObject instances. ssl.wrap_socket() is
also deprecated.
The only test case for direct instantiation was added a couple of days
ago for IDNA testing.
Signed-off-by: Christian Heimes <christian@python.org>
The ssl module function ssl.wrap_socket() has been de-emphasized
and deprecated in favor of the more secure and efficient
SSLContext.wrap_socket() method.
Signed-off-by: Christian Heimes <christian@python.org>
* bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes
Misc fixes and workarounds for compatibility with OpenSSL 1.1.1-pre1 and
TLS 1.3 support. With OpenSSL 1.1.1, Python negotiates TLS 1.3 by
default. Some test cases only apply to TLS 1.2. Other tests currently
fail because the threaded or async test servers stop after failure.
I'm going to address these issues when OpenSSL 1.1.1 reaches beta.
OpenSSL 1.1.1 has added a new option OP_ENABLE_MIDDLEBOX_COMPAT for TLS
1.3. The feature is enabled by default for maximum compatibility with
broken middle boxes. Users should be able to disable the hack and CPython's test suite needs
it to verify default options.
Signed-off-by: Christian Heimes <christian@python.org>
When comprehensions switched to using a nested scope, the old
code for generating a temporary name to hold the accumulation
target became redundant, but was never actually removed.
Patch by Nitish Chandra.
unsafe_hash=False is now the default. It is the same behavior as the old hash=None parameter. unsafe_hash=True will try to add __hash__. If it already exists, TypeError is raised.
The ctypes module used to depend on indirect linking for dlopen. The shared
extension is now explicitly linked against libdl on platforms with dl.
Signed-off-by: Christian Heimes <christian@python.org>
The ssl module now detects missing NPN support in LibreSSL.
Co-Authored-By: Bernard Spil <brnrd@FreeBSD.org>
Signed-off-by: Christian Heimes <christian@python.org>
* The SSLSocket is no longer implemented on top of SSLObject to
avoid an extra level of indirection.
* Owner and session are now handled in the internal constructor.
* _ssl._SSLSocket now uses the same method names as SSLSocket and
SSLObject.
* Channel binding type check is now handled in C code. Channel binding
is always available.
The patch also changes the signature of SSLObject.__init__(). In my
opinion it's fine. A SSLObject is not a user-constructable object.
SSLContext.wrap_bio() is the only valid factory.
ssl.match_hostname() has been simplified and no longer depends on re and
ipaddress module for wildcard and IP addresses. Error reporting for invalid
wildcards has been improved.
Signed-off-by: Christian Heimes <christian@python.org>
Previously, the ssl module stored international domain names (IDNs)
as U-labels. This is problematic for a number of reasons -- for
example, it made it impossible for users to use a different version
of IDNA than the one built into Python.
After this change, we always convert to A-labels as soon as possible,
and use them for all internal processing. In particular, server_hostname
attribute is now an A-label, and on the server side there's a new
sni_callback that receives the SNI servername as an A-label rather than
a U-label.
Make test.support.temp_cwd() fork-safe. The context manager test.support.temp_cwd() no longer removes the temporary directory when executing in a process other than the parent it entered from.
If a forked child exits the context manager it won't do the cleanup.
dump is similar to print but less flexible. lastopenbracketpos is now always initialized in _study2, as was stmt_bracketing, so the class settings are not needed. get_last_open_bracket_pos is never called.
os.dup2() tests for dup3() system call availability at runtime,
but doesn't remember the result across calls, repeating
the test on each call with inheritable=False.
Since the caller of os.dup2() is expected to hold the GIL,
fix this by making the variable holding the test result static.
GUI test test_file_buttons() only looks at initial ascii-only lines,
but failed on systems where open() defaults to 'ascii' because
readline() internally reads and decodes far enough ahead to encounter
a non-ascii character in CREDITS.txt.
fnmatch.translate() no longer produces patterns which contain set
operations.
Sets starting with '[' or containing '--', '&&', '~~' or '||' will
be interpreted differently in regular expressions in future versions.
Currently they emit warnings. fnmatch.translate() now avoids producing
patterns containing such sets by accident.
Fix a rare but potential pre-exec child process deadlock in subprocess on POSIX systems when marking file descriptors inheritable on exec in the child process. This bug appears to have been introduced in 3.4 with the inheritable file descriptors support.
This also changes Python/fileutils.c `set_inheritable` to use the "slow" two `fcntl` syscall path instead of the "fast" single `ioctl` syscall path when asked to be async signal safe (by way of being asked not to raise exceptions). `ioctl` is not a POSIX async-signal-safe approved function.
ref: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
When `__getattr__` is implemented, attribute lookup will always fall back to that,
even if the initial failure comes from `__getattribute__` or a descriptor's `__get__`
method (including property methods).
Modify RE examples in documentation to use raw strings to prevent DeprecationWarning.
Add text to REGEX HOWTO to highlight the deprecation. Approved by Serhiy Storchaka.
In the tutorial about the Generator expression, there is an example with
a dict comprehension and not with a generator expression, just removed
the code.
This issue covers various changes for the macOS installers provided via python.org for 3.7.0.
- Provide a provisional new installer variant for macOS 10.9 and later systems with 64-bit (x86_64) architecture only. Apple has made it known that future versions of macOS will only fully support 64-bit executables and some other third-party software suppliers have chosen 10.9 as their oldest supported system.
- Support **Tcl/Tk 8.6** with the 10.9 installer variant.
- Upgrade **OpenSSL** to 1.1.0g and **SQLite** to 3.22.0.
- The compiler name used for the interpreter build and for modules built with **Distutils / pip** is now _gcc_ rather than _gcc-4.2_. And extension module builds will no longer try to force use of an old SDK if present.
No longer add /Library/Python/3.x/site-packages, the Apple-supplied
system Python site-packages directory, to sys.path for macOS framework
builds in case Apple ships a version of Python 3. A similar change
was made earlier to Python 2.7 where it was found that the coupling
between the system Python and a user-installed framework Python often
caused confusion or pip install failures.
This allows the compression level to be specified when writing zipfiles
(for the entire file *and* overridden on a per-file basis).
Contributed by Bo Bayles
Vlad Starostin
Aaron Hall, MBA
Serhiy Storchaka
Mario Corchero
CtrlZvi
Cheryl Sabella
Ivan Levkivskyi
Terry Jan Reedy
Zvezdan Petkovic
Steve Dower
Nina Zakharenko
Petr Viktorin
Eric V. Smith
Christopher Beacham
Eitan Adler
Christian Heimes
Stéphane Wirtel
Victor Stinner
Matthias Bussonnier
Segev Finer
Ned Deily
Travis DePrato
Michael Lazar
Rolf Eike Beer
Oren Milman
sblondon
Andrés Delfino
Antoine Pitrou
lekma
Michael Selik
Brett Cannon
Matthias Klose
Siddhesh Poyarekar
Dong-hee Na
Zsolt Dollenstein
Pablo Galindo
Łukasz Langa
Chris Bradbury
Berker Peksag
Paul Moore
xdegaye
Gregory P. Smith
Denis Osipov
Aaron Ang
Wolfgang Maier
INADA Naoki
Zackery Spytz
Nick Coghlan
jdemeyer
Mark Nemec
Alex Gaynor
Jay Crotts
Marcel Plch
Raymond Hettinger
Skip Montanaro
hui shang
Julien Palard
Alexey Izbyshev
Garvit Khatri
Xiang Zhang
Himanshu Lakhara
jab
Thomas Moreau
Matt Eaton
aetracht
Miro Hrončok
Antoine Pietri
Nir Soffer
Emanuele Gaifas
Andrew Svetlov
Yury Selivanov
Victor Stinner
Jamie Davis
Jon Wayne Parrott
Joffrey F
Tobotimus
Nitish Chandra
Sergey Fedoseev
animalize
Anselm Kruis
Eddie Elizondo
Paul Price
chason
Éric Araujo
Kyle Altendorf
Bar Harel
Коренберг Марк
SSE4
Alan D Moore
Barry Warsaw
Bernhard M. Wiedemann
John Jolly
Anders Lorentsen
Bo Bayles
Benjamin Peterson