The urllib.robotparser's __str__ representation now includes wildcard
entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra
newlines that were being appended to the end of the string.
The hash implementation casts the input pointer to uint64_t* and directly reads
from this, which may cause unaligned accesses. Use memcpy() instead so this code
will not crash with SIGBUS on sparc.
https://bugs.gentoo.org/show_bug.cgi?id=636400
Remove the paragraph where we explain that os.utime() does not support a
directory as path under Windows. Patch by Jan-Philip Gehrcke
Co-authored-by: Jan-Philip Gehrcke <jgehrcke@gmail.com>
In text and entry boxes, this affects selection by double-click,
movement left/right by control-left/right, and deletion left/right
by control-BACKSPACE/DEL.
METH_NOARGS functions need only a single argument but they are cast
into a PyCFunction, which takes two arguments. This triggers an
invalid function cast warning in gcc8 due to the argument mismatch.
Fix this by adding a dummy unused argument.
External importers were being added in both phases of the import
system initialisation.
They're only supposed to be added in the second phase, after the
import machinery has been appropriately configured.
* bpo-33251: ConfigParser.items no longer returns items present in vars.
Documentation for `ConfigParser.items()` states:
'Items present in vars no longer appear in the result.'
This fix aligns behaviour to that specified in the documentation.
Starting with glibc 2.27.9000-xxx, sigaddset() can return EINVAL for some
reserved signal numbers between 1 and NSIG. The `range(1, NSIG)` idiom
is commonly used to select all signals for blocking with `pthread_sigmask`.
So we ignore the sigaddset() return value until we expose sigfillset()
to provide a better idiom.
Setup modules are no longer built with -DPy_BUILD_CORE by default,
as using that flag may now require including additional internal-only header files.
Instead, only the modules that specifically need it use that setting.
Fix clang ubsan (undefined behavior sanitizer) warnings in dictobject.c by
adjusting how the internal struct _dictkeysobject shared keys structure is
declared.
This remains ABI compatible. We get rid of the union at the end of the
struct being used for conveinence to avoid typecasting in favor of char[]
variable length array at the end of a struct. This is known to clang to be
used for variable sized objects and will not cause an undefined behavior
problem. Similarly, char arrays do not have strict aliasing undefined
behavior when cast.
PEP-007 does not currently list variable length arrays (VLAs) as allowed
in our subset of C99. If this turns out to be a problem, the fix to this is
to change the char `dk_indices[]` into `dk_indices[1]` and restore the
three size computation subtractions this change removes:
`- Py_MEMBER_SIZE(PyDictKeysObject, dk_indices)`
If this works as is I'll make a separate PR to update PEP-007.
* Now uses pickle protocol 4
* Doesn't wrap the grammar's `__dict__` in ordered dictionaries anymore as
dictionaries in Python 3.6+ are ordered by default
This still produces deterministic pickles (that hash the same with MD5).
Tested with different PYTHONHASHSEED values.
The pydoc CLI assumed -m pydoc would add the empty string
to sys.path, and hence got confused when it switched to
adding the full initial working directory instead.
This refactors the pydoc CLI path manipulation to be
more testable, and ensures it won't accidentally
remove the standard library directory containing
pydoc itself from sys.path.
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
* bpo-29613: Added support for SameSite cookies
Implemented as per draft
https://tools.ietf.org/html/draft-west-first-party-cookies-07
* Documented SameSite
And suggestions by members.
* Missing space :(
* Updated News and contributors
* Added version changed details.
* Fix in documentation
* fix in documentation
* Clubbed test cases for same attribute into single.
* Updates
* Style nits + expand tests
* review feedback
* bpo-33201: Modernize "Extension types" doc
* Split tutorial and other topics
* Some small fixes
* Address some review comments
* Rename noddy* to custom* and shoddy to sublist
* Fix markup
This makes performance better and produces shorter pickles. This change is backwards compatible up to the oldest currently supported version of Python (3.4).
Build and link with private copy of Tcl/Tk 8.6 for the macOS 10.6+
installer. The 10.9+ installer variant already does this. This means that
the Python 3.7 provided by the python.org macOS installers no longer need or
use any external versions of Tcl/Tk, either system-provided or user-
installed, such as ActiveTcl.
test_asyncio hangs indefinitely on macOS 10.13.2+ on `read_pty_output()`
using the KqueueSelector. Closing `proto.transport` (as is done in
`write_pty_output()`) seems to fix it.
(cherry picked from commit 12f74d8608)
Co-authored-by: Nathan Henrie <n8henrie@users.noreply.github.com>
Also, re-enable test_read_pty_output on macOS.
This also fixespython/typing#512
This also fixespython/typing#511
As was discussed in both issues, some typing forms deserve to be treated
as immutable by copy and pickle modules, so that:
* copy(X) is X
* deepcopy(X) is X
* loads(dumps(X)) is X # pickled by reference
This PR adds such behaviour to:
* Type variables
* Special forms like Union, Any, ClassVar
* Unsubscripted generic aliases to containers like List, Mapping, Iterable
This not only resolves inconsistencies mentioned in the issues, but also
improves backwards compatibility with previous versions of Python
(including 3.6).
Note that this requires some dances with __module__ for type variables
(similar to NamedTuple) because the class TypeVar itself is define in typing,
while type variables should get module where they were defined.
https://bugs.python.org/issue32873
bpo-32844: subprocess: Fix a potential misredirection of a low fd to stderr.
When redirecting, subprocess attempts to achieve the following state:
each fd to be redirected to is less than or equal to the fd
it is redirected from, which is necessary because redirection
occurs in the ascending order of destination descriptors.
It fails to do so in a couple of corner cases,
for example, if 1 is redirected to 2 and 0 is closed in the parent.
Historically, -m added the empty string as sys.path
zero, meaning it resolved imports against the current
working directory, the same way -c and the interactive
prompt do.
This changes the sys.path initialisation to add the
*starting* working directory as sys.path[0] instead,
such that changes to the working directory while the
program is running will have no effect on imports
when using the -m switch.
- new test case for pre-initialization of sys.warnoptions and sys._xoptions
- restored ability to call these APIs prior to Py_Initialize
- updated the docs for the affected APIs to make it clear they can be
called before Py_Initialize
- also enhanced the existing embedding test cases
to check for expected settings in the sys module
Harden ssl module against LibreSSL CVE-2018-8970.
X509_VERIFY_PARAM_set1_host() is called with an explicit namelen. A new test
ensures that NULL bytes are not allowed.
Signed-off-by: Christian Heimes <christian@python.org>
LibreSSL 2.7 introduced OpenSSL 1.1.0 API. The ssl module now detects
LibreSSL 2.7 and only provides API shims for OpenSSL < 1.1.0 and
LibreSSL < 2.7.
Documentation updates and fixes for failing tests will be provided in
another patch set.
Signed-off-by: Christian Heimes <christian@python.org>
* Added new opcode END_ASYNC_FOR.
* Setting global StopAsyncIteration no longer breaks "async for" loops.
* Jumping into an "async for" loop is now disabled.
* Jumping out of an "async for" loop no longer corrupts the stack.
* Simplify the compiler.
If a non-dataclass derives from a frozen dataclass, allow attributes to be set.
Require either all of the dataclasses in a class hierarchy to be frozen, or all non-frozen.
Store `@dataclass` parameters on the class object under `__dataclass_params__`. This is needed to detect frozen base classes.
Multi-phase initialized modules allow m_traverse to be called while the
module is still being initialized, so module authors may need to account
for that.
Creating backup files with ~ suffix can be undesirable in some environment,
such as when building RPM packages. Instead of requiring the user to remove
those files manually, option -n was added, that simply disables this feature.
-n was selected because 2to3 has the same option with this behavior.
New tests also added.
I also made the comments in line with the builtin Grammar/Grammar. PEP 306 was
withdrawn, Kees Blom's railroad program has been lost to the sands of time for
at least 16 years now (I found a python-dev post from people looking for it).
fstat may block for long time if the file descriptor is on a
non-responsive NFS server, hanging all threads. Most fstat() calls are
handled by _Py_fstat(), releasing the GIL internally, but but
_Py_fstat_noraise() does not release the GIL, and most calls release the
GIL explicitly around it.
This patch fixes last 2 calls to _Py_fstat_no_raise(), avoiding hangs
when calling:
- mmap.mmap()
- os.urandom()
- random.seed()
In some conditions the standard streams will be None or closed in the child process (for example if using "pythonw" instead of "python" on Windows). Avoid failing with a non-0 exit code in those conditions.
Report and initial patch by poxthegreat.
Fix a crash on fork when using a custom memory allocator (ex: using
PYTHONMALLOC env var).
_PyGILState_Reinit() and _PyInterpreterState_Enable() now use the
default RAW memory allocator to allocate a new interpreters mutex on
fork.
Like Python, IDLE optionally runs one startup file in the Shell window
before presenting the first interactive input prompt. For IDLE,
option -s runs a file named in environmental variable IDLESTARTUP or
PYTHONSTARTUP; -r file runs file. Python sets __file__ to the startup
file name before running the file and unsets it before the first
prompt. IDLE now does the same when run normally, without the -n
option.
* Prevent low-grade poplib REDOS (CVE-2018-1060)
The regex to test a mail server's timestamp is susceptible to
catastrophic backtracking on long evil responses from the server.
Happily, the maximum length of malicious inputs is 2K thanks
to a limit introduced in the fix for CVE-2013-1752.
A 2KB evil response from the mail server would result in small slowdowns
(milliseconds vs. microseconds) accumulated over many apop calls.
This is a potential DOS vector via accumulated slowdowns.
Replace it with a similar non-vulnerable regex.
The new regex is RFC compliant.
The old regex was non-compliant in edge cases.
* Prevent difflib REDOS (CVE-2018-1061)
The default regex for IS_LINE_JUNK is susceptible to
catastrophic backtracking.
This is a potential DOS vector.
Replace it with an equivalent non-vulnerable regex.
Also introduce unit and REDOS tests for difflib.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
OpenSSL 1.1 has introduced a new API to set the minimum and maximum
supported protocol version. The API is easier to use than the old
OP_NO_TLS1 option flags, too.
Since OpenSSL has no call to set minimum version to highest supported,
the implementation emulate maximum_version = MINIMUM_SUPPORTED and
minimum_version = MAXIMUM_SUPPORTED by figuring out the minumum and
maximum supported version at compile time.
Signed-off-by: Christian Heimes <christian@python.org>
Direct instantiation of SSLSocket and SSLObject objects is now prohibited.
The constructors were never documented, tested, or designed as public
constructors. The SSLSocket constructor had limitations. For example it was
not possible to enabled hostname verification except was
ssl_version=PROTOCOL_TLS_CLIENT with cert_reqs=CERT_REQUIRED.
SSLContext.wrap_socket() and SSLContext.wrap_bio are the recommended API
to construct SSLSocket and SSLObject instances. ssl.wrap_socket() is
also deprecated.
The only test case for direct instantiation was added a couple of days
ago for IDNA testing.
Signed-off-by: Christian Heimes <christian@python.org>
The ssl module function ssl.wrap_socket() has been de-emphasized
and deprecated in favor of the more secure and efficient
SSLContext.wrap_socket() method.
Signed-off-by: Christian Heimes <christian@python.org>
* bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes
Misc fixes and workarounds for compatibility with OpenSSL 1.1.1-pre1 and
TLS 1.3 support. With OpenSSL 1.1.1, Python negotiates TLS 1.3 by
default. Some test cases only apply to TLS 1.2. Other tests currently
fail because the threaded or async test servers stop after failure.
I'm going to address these issues when OpenSSL 1.1.1 reaches beta.
OpenSSL 1.1.1 has added a new option OP_ENABLE_MIDDLEBOX_COMPAT for TLS
1.3. The feature is enabled by default for maximum compatibility with
broken middle boxes. Users should be able to disable the hack and CPython's test suite needs
it to verify default options.
Signed-off-by: Christian Heimes <christian@python.org>
When comprehensions switched to using a nested scope, the old
code for generating a temporary name to hold the accumulation
target became redundant, but was never actually removed.
Patch by Nitish Chandra.
unsafe_hash=False is now the default. It is the same behavior as the old hash=None parameter. unsafe_hash=True will try to add __hash__. If it already exists, TypeError is raised.
The ctypes module used to depend on indirect linking for dlopen. The shared
extension is now explicitly linked against libdl on platforms with dl.
Signed-off-by: Christian Heimes <christian@python.org>
The ssl module now detects missing NPN support in LibreSSL.
Co-Authored-By: Bernard Spil <brnrd@FreeBSD.org>
Signed-off-by: Christian Heimes <christian@python.org>
* The SSLSocket is no longer implemented on top of SSLObject to
avoid an extra level of indirection.
* Owner and session are now handled in the internal constructor.
* _ssl._SSLSocket now uses the same method names as SSLSocket and
SSLObject.
* Channel binding type check is now handled in C code. Channel binding
is always available.
The patch also changes the signature of SSLObject.__init__(). In my
opinion it's fine. A SSLObject is not a user-constructable object.
SSLContext.wrap_bio() is the only valid factory.
ssl.match_hostname() has been simplified and no longer depends on re and
ipaddress module for wildcard and IP addresses. Error reporting for invalid
wildcards has been improved.
Signed-off-by: Christian Heimes <christian@python.org>
Previously, the ssl module stored international domain names (IDNs)
as U-labels. This is problematic for a number of reasons -- for
example, it made it impossible for users to use a different version
of IDNA than the one built into Python.
After this change, we always convert to A-labels as soon as possible,
and use them for all internal processing. In particular, server_hostname
attribute is now an A-label, and on the server side there's a new
sni_callback that receives the SNI servername as an A-label rather than
a U-label.
dump is similar to print but less flexible. lastopenbracketpos is now always initialized in _study2, as was stmt_bracketing, so the class settings are not needed. get_last_open_bracket_pos is never called.
os.dup2() tests for dup3() system call availability at runtime,
but doesn't remember the result across calls, repeating
the test on each call with inheritable=False.
Since the caller of os.dup2() is expected to hold the GIL,
fix this by making the variable holding the test result static.
GUI test test_file_buttons() only looks at initial ascii-only lines,
but failed on systems where open() defaults to 'ascii' because
readline() internally reads and decodes far enough ahead to encounter
a non-ascii character in CREDITS.txt.
fnmatch.translate() no longer produces patterns which contain set
operations.
Sets starting with '[' or containing '--', '&&', '~~' or '||' will
be interpreted differently in regular expressions in future versions.
Currently they emit warnings. fnmatch.translate() now avoids producing
patterns containing such sets by accident.
Fix a rare but potential pre-exec child process deadlock in subprocess on POSIX systems when marking file descriptors inheritable on exec in the child process. This bug appears to have been introduced in 3.4 with the inheritable file descriptors support.
This also changes Python/fileutils.c `set_inheritable` to use the "slow" two `fcntl` syscall path instead of the "fast" single `ioctl` syscall path when asked to be async signal safe (by way of being asked not to raise exceptions). `ioctl` is not a POSIX async-signal-safe approved function.
ref: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
When `__getattr__` is implemented, attribute lookup will always fall back to that,
even if the initial failure comes from `__getattribute__` or a descriptor's `__get__`
method (including property methods).
Modify RE examples in documentation to use raw strings to prevent DeprecationWarning.
Add text to REGEX HOWTO to highlight the deprecation. Approved by Serhiy Storchaka.
In the tutorial about the Generator expression, there is an example with
a dict comprehension and not with a generator expression, just removed
the code.
This issue covers various changes for the macOS installers provided via python.org for 3.7.0.
- Provide a provisional new installer variant for macOS 10.9 and later systems with 64-bit (x86_64) architecture only. Apple has made it known that future versions of macOS will only fully support 64-bit executables and some other third-party software suppliers have chosen 10.9 as their oldest supported system.
- Support **Tcl/Tk 8.6** with the 10.9 installer variant.
- Upgrade **OpenSSL** to 1.1.0g and **SQLite** to 3.22.0.
- The compiler name used for the interpreter build and for modules built with **Distutils / pip** is now _gcc_ rather than _gcc-4.2_. And extension module builds will no longer try to force use of an old SDK if present.
No longer add /Library/Python/3.x/site-packages, the Apple-supplied
system Python site-packages directory, to sys.path for macOS framework
builds in case Apple ships a version of Python 3. A similar change
was made earlier to Python 2.7 where it was found that the coupling
between the system Python and a user-installed framework Python often
caused confusion or pip install failures.
This allows the compression level to be specified when writing zipfiles
(for the entire file *and* overridden on a per-file basis).
Contributed by Bo Bayles
Do not allow receiving a SIGINT to cause the subprocess module to trigger an
immediate SIGKILL of the child process. SIGINT is normally sent to all child
processes by the OS at the same time already as was the established normal
behavior in 2.7 and 3.2. This behavior change was introduced during the fix to https://bugs.python.org/issue12494 and is generally surprising to command line
tool users who expect other tools launched in child processes to get their own
SIGINT and do their own cleanup.
In Python 3.3-3.6 subprocess.call and subprocess.run would immediately
SIGKILL the child process upon receiving a SIGINT (which raises a
KeyboardInterrupt). We now give the child a small amount of time to
exit gracefully before resorting to a SIGKILL.
This is also the case for subprocess.Popen.__exit__ which would
previously block indefinitely waiting for the child to die. This was
hidden from many users by virtue of subprocess.call and subprocess.run
sending the signal immediately.
Behavior change: subprocess.Popen.__exit__ will not block indefinitely
when the exiting exception is a KeyboardInterrupt. This is done for
user friendliness as people expect their ^C to actually happen. This
could cause occasional orphaned Popen objects when not using `call` or
`run` with a child process that hasn't exited.
Refactoring involved: The Popen.wait method deals with the
KeyboardInterrupt second chance, existing platform specific internals
have been renamed to _wait().
Also fixes comment typos.
Fix socket(fileno=fd) by auto-detecting the socket's family, type,
and proto from the file descriptor. The auto-detection can be overruled
by passing in family, type, and proto explicitly.
Without the fix, all socket except for TCP/IP over IPv4 are basically broken:
>>> s = socket.create_connection(('www.python.org', 443))
>>> s
<socket.socket fd=3, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=6, laddr=('2003:58:bc4a:3b00:56ee:75ff:fe47:ca7b', 59730, 0, 0), raddr=('2a04:4e42:1b::223', 443, 0, 0)>
>>> socket.socket(fileno=s.fileno())
<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('2003:58:bc4a:3b00::%2550471192', 59730, 0, 2550471192), raddr=('2a04:4e42:1b:0:700c:e70b:ff7f:0%2550471192', 443, 0, 2550471192)>
Signed-off-by: Christian Heimes <christian@python.org>
The commit removes one unnecessary "if" clause in genobject.c. That "if" clause was masking un-awaited coroutines warnings just to make writing unittests more convenient.
Add test cases for IDNA 2003 and 2008 host names. IDNA 2003
internationalized host names are working since bpo-31399 has landed. IDNA
2008 deviations are still broken and will be fixed in another patch.
Signed-off-by: Christian Heimes <christian@python.org>
Until now Python used a hard coded white list of default TLS cipher
suites. The old approach has multiple downsides. OpenSSL's default
selection was completely overruled. Python did neither benefit from new
cipher suites (ChaCha20, TLS 1.3 suites) nor blacklisted cipher suites.
For example we used to re-enable 3DES.
Python now defaults to OpenSSL DEFAULT cipher suite selection and black
lists all unwanted ciphers. Downstream vendors can override the default
cipher list with --with-ssl-default-suites.
Signed-off-by: Christian Heimes <christian@python.org>
Better account for single-line compound statements and
semi-colon separated statements when suggesting
Py3 replacements for Py2 print statements.
Initial patch by Nitish Chandra.
Class authors no longer need to specify repr=False if they want to provide a custom __repr__ for dataclasses. The same thing applies for the other dunder methods that the dataclass decorator adds. If dataclass finds that a dunder methods is defined in the class, it will not overwrite it.
bpo-31399: Let OpenSSL verify hostname and IP
The ssl module now uses OpenSSL's X509_VERIFY_PARAM_set1_host() and
X509_VERIFY_PARAM_set1_ip() API to verify hostname and IP addresses.
* Remove match_hostname calls
* Check for libssl with set1_host, libssl must provide X509_VERIFY_PARAM_set1_host()
* Add documentation for OpenSSL 1.0.2 requirement
* Don't support OpenSSL special mode with a leading dot, e.g. ".example.org" matches "www.example.org". It's not standard conform.
* Add hostname_checks_common_name
Signed-off-by: Christian Heimes <christian@python.org>
The hmac module now has hmac.digest(), which provides an optimized HMAC
digest for short messages. hmac.digest() is up to three times faster
than hmac.HMAC().digest().
Signed-off-by: Christian Heimes <christian@python.org>
dictview_repr(): Use a Py_ReprEnter() / Py_ReprLeave() pair to check
for recursion, and produce "..." if so.
test_recursive_repr(): Check for the string rather than a
RecursionError. (Test cannot be any tighter as contents are
implementation-dependent.)
test_deeply_nested_repr(): Add new test, replacing the original
test_recursive_repr(). It checks that a RecursionError is raised in
the case of a non-recursive but deeply nested structure. (Very
similar to what test_repr_deep() in test/test_dict.py does for a
normal dict.)
OrderedDictTests: Add new test case, to test behavior on OrderedDict
instances containing their own values() or items().
* bpo-32662: Implement Server.start_serving() and Server.serve_forever()
New methods:
* Server.start_serving(),
* Server.serve_forever(), and
* Server.is_serving().
Add 'start_serving' keyword parameter to loop.create_server() and
loop.create_unix_server().
* Add timezone to datetime C API
* Add documentation for timezone C API macros
* Add dedicated tests for datetime type check macros
* Remove superfluous C API test
* Drop support for TimeZoneType in datetime C API
* Expose UTC singleton to the datetime C API
* Update datetime C-API documentation to include links
* Add reference count information for timezone constructors
To support reproducible builds, the setting of of SOURCE_DATE_EPOCH triggers the py_compile module -- and by extension, compileall -- to forcibly compile with hash-based .pyc files. This eliminates the possibility of timestamp-based .pyc files which vary between builds.
Specifically, it's not possible to subclass Task/Future classes
and override the following methods:
* Future._schedule_callbacks
* Task._step
* Task._wakeup
glibc is deprecating libcrypt in favor of libxcrypt, however python assumes
that crypt.h will always be included. This change makes the header inclusion
explicit when libxcrypt is present on the system.
* bpo-31801: Enum: add _ignore_ as class option
_ignore_ is a list, or white-space seperated str, of names that will not
be candidates for members; these names, and _ignore_ itself, are removed
from the final class.
* bpo-31801: Enum: add documentation for _ignore_
* bpo-31801: Enum: remove trailing whitespace
* bpo-31801: Enum: fix bulleted list format
* bpo-31801: add version added for _ignore_
* Add coro.cr_origin and sys.set_coroutine_origin_tracking_depth
* Use coroutine origin information in the unawaited coroutine warning
* Stop using set_coroutine_wrapper in asyncio debug mode
* In BaseEventLoop.set_debug, enable debugging in the correct thread
Add test certs and test for ECDSA cert and EC/RSA dual mode.
I'm also adding certs for IDNA 2003/2008 tests and simplify some test
data handling.
Signed-off-by: Christian Heimes <christian@python.org>
Add https://www.gnu.org/software/autoconf-archive/ax_check_openssl.html
to auto-detect compiler flags, linker flags and libraries to compile
OpenSSL extensions. The M4 macro uses pkg-config and falls back to
manual detection.
Add autoconf magic to detect usable X509_VERIFY_PARAM_set1_host()
and related functions.
Refactor setup.py to use new config vars to compile _ssl and _hashlib
modules.
Signed-off-by: Christian Heimes <christian@python.org>
The suggested replacement for print statements previously failed to account
for leading whitespace and hence could end up including unwanted text in
the proposed call to the print builtin.
Patch by Sanyam Khurana.
* Add tests for date subclass alternate constructors
* Switch over alternate date constructors to fast path
* Switch datetime constructors to fastpath, fix bpo-32404
* Add fast path for datetime in date subclass constructor
* Set fold in constructor in datetime.combine
* Add news entries.
AttributeError was raised always when attribute is not found.
This commit skip raising AttributeError when `tp_getattro` is `PyObject_GenericGetAttr`.
It makes hasattr() and getattr() about 4x faster when attribute is not found.
* Add _Py_GetLocaleconvNumeric() function: decode decimal_point and
thousands_sep fields of localeconv() from the LC_NUMERIC encoding,
rather than decoding from the LC_CTYPE encoding.
* Modify locale.localeconv() and "n" formatter of str.format() (for
int, float and complex to use _Py_GetLocaleconvNumeric()
internally.
PyMemoryView_FromMemory() created a memoryview referring to
the internal data of the string. When the string is destroyed
the memoryview become referring to a freed memory.
- primary change is to add a new default filter entry for
'default::DeprecationWarning:__main__'
- secondary change is an internal one to cope with plain
strings in the warning module's internal filter list
(this avoids the need to create a compiled regex object
early on during interpreter startup)
- assorted documentation updates, including many more
examples of configuring the warnings settings
- additional tests to ensure that both the pure Python and
the C accelerated warnings modules have the expected
default configuration
Third party projects may wish to hide their own internal machinery in
order to present more comprehensible tracebacks to end users
(e.g. Jinja2 and Trio both do this).
Previously such projects have had to rely on ctypes to do so:
fe3dadacdf/jinja2/debug.py (L345)1e86b1aee8/trio/_core/_multierror.py (L296)
This provides a Python level API for creating and modifying real
Traceback objects, allowing tracebacks to be edited at runtime.
Patch by Nathaniel Smith.
The picklers do no longer allocate temporary memory when dumping large
bytes and str objects into a file object. Instead the data is
directly streamed into the underlying file object.
Previously the C implementation would buffer all content and issue a
single call to file.write() at the end of the dump. With protocol 4
this behavior has changed to issue one call to file.write() per frame.
The Python pickler with protocol 4 now dumps each frame content as a
memoryview to an IOBytes instance that is never reused and the
memoryview is no longer released after the call to write. This makes it
possible for the file object to delay access to the memoryview of
previous frames without forcing any additional memory copy as was
already possible with the C pickler.
Add a new argument "-m" to the pdb module to allow
users to run `python -m pdb -m my_module_name`.
This relies on private APIs in the runpy module to work,
but we can get away with that since they're both part of
the standard library and can be updated together if
the runpy internals get refactored.
This module has never been enabled by default, never worked correctly
on x86-64, and caused ABI problems that caused C extension
compatibility. See bpo-29137 for details/discussion.
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
Starting with AIX6.1 there is support in libc.a for uuid (RFC4122)
This patch provides the changes needed for this integration with the OS.
On AIX the base function is uuid_create() rather than uuid_generate_time()
The AIX uuid_t typedef is more aligned to the UUID field based definition
while the Linux typedef that is more aligned with UUID bytes
(or perhaps UUID bytes_le) definitions.
This is more complicated than it should be because we need to preserve the
useful mtime-based regeneration feature that lib2to3.pgen2.driver.load_grammar
has. We only look for the pickled grammar file with pkgutil.get_data and only if
the source file does not exist.
* _Py_InitializeCore() doesn't call _PyMem_SetupAllocators() anymore
if the PYTHONMALLOC environment variable is not set.
* pymain_cmdline() now sets the allocator to the default, instead of
setting the allocator in subfunctions.
* Py_SetStandardStreamEncoding() now calls
_PyMem_SetDefaultAllocator() to get a known allocator, to be able
to release the memory with the same allocator.
Even though Python marks any handles it opens as non-inheritable there
is still a race when using `subprocess.Popen` since creating a process
with redirected stdio requires temporarily creating inheritable handles.
By implementing support for `subprocess.Popen(close_fds=True)` we fix
this race.
In order to implement this we use PROC_THREAD_ATTRIBUTE_HANDLE_LIST
which is available since Windows Vista. Which allows to pass an explicit
list of handles to inherit when creating a process.
This commit also adds `STARTUPINFO.lpAttributeList["handle_list"]`
which can be used to control PROC_THREAD_ATTRIBUTE_HANDLE_LIST
directly.
Exactly which locale requests will end up giving
you the "C" locale is actually platform dependent.
A blank locale and "POSIX" will translate to "C"
on most Linux distros, but may not do so on other platforms, so this adjusts the way the tests are structured to better account for that.
This is an initial step towards fixing the current
test failure on Cygwin (hence the issue reference)
It no longer spends much time doing complex calculations and no
longer consumes much memory for creating large constants that will
be dropped later.
This fixes also bpo-21074.
bpo-32329, bpo-32030:
* The -R option now turns on hash randomization when the
PYTHONHASHSEED environment variable is set to 0 Previously, the
option was ignored.
* sys.flags.hash_randomization is now properly set to 0 when hash
randomization is turned off by PYTHONHASHSEED=0.
* _PyCoreConfig_ReadEnv() now reads the PYTHONHASHSEED environment
variable. _Py_HashRandomization_Init() now only apply the
configuration, it doesn't read PYTHONHASHSEED anymore.
asyncio.get_event_loop(), and, subsequently asyncio._get_running_loop()
are one of the most frequently executed functions in asyncio. They also
can't be sped up by third-party event loops like uvloop.
When implemented in C they become 4x faster.
* Add -X utf8 command line option, PYTHONUTF8 environment variable
and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
_winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
mode. As a side effect, open() now uses the UTF-8 encoding by
default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
always copy flag values, rather than only copying if the new value
is greater than the old value.
Rather than supporting dev mode directly in the warnings module, this
instead adjusts the initialisation code to add an extra 'default'
entry to sys.warnoptions when dev mode is enabled.
This ensures that dev mode behaves *exactly* as if `-Wdefault` had
been passed on the command line, including in the way it interacts
with `sys.warnoptions`, and with other command line flags like `-bb`.
Fix also bpo-20361: have -b & -bb options take precedence over any
other warnings options.
Patch written by Nick Coghlan, with minor modifications of Victor Stinner.
The error messages in `object.__new__` and `object.__init__` now aim
to point the user more directly at the name of the class being instantiated
in cases where they *haven't* been overridden (on the assumption that
the actual problem is a missing `__new__` or `__init__` definition in the
class body).
When they *have* been overridden, the errors still report themselves as
coming from object, on the assumption that the problem is with the call
up to the base class in the method implementation, rather than with the
way the constructor is being called.
Reference siphash takes the keys as a bytes, so it makes sense to byte swap
when reifying the keys as 64-bit integers. However, Python's siphash takes host
integers in to start with.
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.
While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:
- The core changes to importlib to understand how to read, validate, and
regenerate hash-based pycs.
- Support for generating hash-based pycs in py_compile and compileall.
- Modifications to our siphash implementation to support passing a custom
key. We then expose it to importlib through _imp.
- Updates to all places in the interpreter, standard library, and tests that
manually generate or parse pyc files to grok the new format.
- Support in the interpreter command line code for long options like
--check-hash-based-pycs.
- Tests and documentation for all of the above.
* Convert asyncio/tasks.py to async/await
* Convert asyncio/queues.py to async/await
* Convert asyncio/test_utils.py to async/await
* Convert asyncio/base_subprocess.py to async/await
* Convert asyncio/subprocess.py to async/await
* Convert asyncio/streams.py to async/await
* Fix comments
* Convert asyncio/locks.py to async/await
* Convert asyncio.sleep to async def
* Add a comment
* Add missing news
* Convert stubs from AbstrctEventLoop to async functions
* Convert subprocess_shell/subprocess_exec
* Convert connect_read_pipe/connect_write_pip to async/await syntax
* Convert create_datagram_endpoint
* Convert create_unix_server/create_unix_connection
* Get rid of old style coroutines in unix_events.py
* Convert selector_events.py to async/await
* Convert wait_closed and create_connection
* Drop redundant line
* Convert base_events.py
* Code cleanup
* Drop redundant comments
* Fix indentation
* Add explicit tests for compatibility between old and new coroutines
* Convert windows event loop to use async/await
* Fix double awaiting of async function
* Convert asyncio/locks.py
* Improve docstring
* Convert tests to async/await
* Convert more tests
* Convert more tests
* Convert more tests
* Convert tests
* Improve test
* Rather than raise TypeError, warn and call list() on the value.
* Fix tests, revise NEWS and whatsnew text.
* Revise documentation, a string is okay as well.
* Ensure 'requires' and 'obsoletes' are real lists.
* Test that requires and obsoletes are turned to lists.
When tk event handling is driven by IDLE's run loop, a confusing
and distracting queue.EMPTY traceback context is no longer added
to tk event exception tracebacks. The traceback is now the same
as when event handling is driven by user code. Patch based on
a suggestion by Serhiy Storchaka.
The original algorithm tried to delegate the folding to the tokens so
that those tokens whose folding rules differed could specify the
differences. However, this resulted in a lot of duplicated code because
most of the rules were the same.
The new algorithm moves all folding logic into a set of functions
external to the token classes, but puts the information about which
tokens can be folded in which ways on the tokens...with the exception of
mime-parameters, which are a special case (which was not even
implemented in the old folder).
This algorithm can still probably be improved and hopefully simplified
somewhat.
Note that some of the test expectations are changed. I believe the
changes are toward more desirable and consistent behavior: in general
when (re) folding a line the canonical version of the tokens is
generated, rather than preserving errors or extra whitespace.
Travis DePrato
Eric V. Smith
Michael Lazar
Rolf Eike Beer
Oren Milman
sblondon
Ivan Levkivskyi
Serhiy Storchaka
Andrés Delfino
Antoine Pitrou
lekma
Ned Deily
Stéphane Wirtel
Michael Selik
Brett Cannon
Matthias Klose
Terry Jan Reedy
Siddhesh Poyarekar
Dong-hee Na
Zsolt Dollenstein
Pablo Galindo
Cheryl Sabella
Chris Bradbury
Berker Peksag
Paul Moore
xdegaye
Gregory P. Smith
Denis Osipov
Aaron Ang
Wolfgang Maier
INADA Naoki
Łukasz Langa
Zackery Spytz
Nick Coghlan
Steve Dower
jdemeyer
Mark Nemec
Alex Gaynor
Jay Crotts
Marcel Plch
Raymond Hettinger
hui shang
Julien Palard
Alexey Izbyshev
Garvit Khatri
Christian Heimes
Xiang Zhang
Himanshu Lakhara
jab
Thomas Moreau
Matt Eaton
aetracht
Miro Hrončok
Antoine Pietri
Nir Soffer
Emanuele Gaifas
Andrew Svetlov
Yury Selivanov
Victor Stinner
Jamie Davis
Jon Wayne Parrott
Joffrey F
Tobotimus
Nitish Chandra
Sergey Fedoseev
animalize
Eddie Elizondo
Paul Price
chason
Éric Araujo
Kyle Altendorf
Bar Harel
Segev Finer
Коренберг Марк
SSE4
Alan D Moore
Mario Corchero
Barry Warsaw
Bernhard M. Wiedemann
John Jolly
Anders Lorentsen
Bo Bayles
Benjamin Peterson
Eric Snow
Mark Shannon
jayyyin
embg
larryhastings
jcea
Mads Jensen
bennorth
mwidjaja
José Melero Fernández
Ilya Kulakov
Paul Ganssle
stratakis
Ethan Furman
Nathaniel J. Smith
Sanyam Khurana
yahya-abou-imran
David Carlier
Max Bélanger
Olivier Grisel
Michael Felt
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి)
Neil Aspinall
Julien Duponchelle
Giuseppe Scrivano
Anthony Sottile
Jelle Zijlstra
Licht Takeuchi
Neil Schemenauer
Mariatta
R. David Murray