In the limited C API, PyObject_INIT() and PyObject_INIT_VAR() are now
defined as aliases to PyObject_Init() and PyObject_InitVar() to make
their implementation opaque. It avoids to leak implementation details
in the limited C API.
Exclude the following functions from the limited C API, move them
from object.h to cpython/object.h:
* _Py_NewReference()
* _Py_ForgetReference()
* _PyTraceMalloc_NewReference()
* _Py_GetRefTotal()
Exclude trashcan mechanism from the limited C API: it requires access to
PyTypeObject and PyThreadState structure fields, whereas these structures
are opaque in the limited C API.
The trashcan mechanism never worked with the limited C API. Move it
from object.h to cpython/object.h.
* bpo-39491: Merge PEP 593 (typing.Annotated) support
PEP 593 has been accepted some time ago. I got a green light for merging
this from Till, so I went ahead and combined the code contributed to
typing_extensions[1] and the documentation from the PEP 593 text[2].
My changes were limited to:
* removing code designed for typing_extensions to run on older Python
versions
* removing some irrelevant parts of the PEP text when copying it over as
documentation and otherwise changing few small bits to better serve
the purpose
* changing the get_type_hints signature to match reality (parameter
names)
I wasn't entirely sure how to go about crediting the authors but I used
my best judgment, let me know if something needs changing in this
regard.
[1] 8280de241f/typing_extensions/src_py3/typing_extensions.py
[2] 17710b8798/pep-0593.rst
When called on a closed object, readinto() segfaults on account
of a write to a freed buffer:
==220553== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==220553== Access not within mapped region at address 0x2A
==220553== at 0x48408A0: memmove (vg_replace_strmem.c:1272)
==220553== by 0x58DB0C: _buffered_readinto_generic (bufferedio.c:972)
==220553== by 0x58DCBA: _io__Buffered_readinto_impl (bufferedio.c:1053)
==220553== by 0x58DCBA: _io__Buffered_readinto (bufferedio.c.h:253)
Reproducer:
reader = open ("/dev/zero", "rb")
_void = reader.read (42)
reader.close ()
reader.readinto (bytearray (42)) ### BANG!
The problem exists since 2012 when commit dc469454ec added code
to free the read buffer on close().
Signed-off-by: Philipp Gesang <philipp.gesang@intra2net.com>
Currently, during runtime destruction, `_PyImport_Cleanup` is clearing the interpreter state before clearing out the modules themselves. This leads to a segfault on modules that rely on the module state to clear themselves up.
For example, let's take the small snippet added in the issue by @DinoV :
```
import _struct
class C:
def __init__(self):
self.pack = _struct.pack
def __del__(self):
self.pack('I', -42)
_struct.x = C()
```
The module `_struct` uses the module state to run `pack`. Therefore, the module state has to be alive until after the module has been cleared out to successfully run `C.__del__`. This happens at line 606, when `_PyImport_Cleanup` calls `_PyModule_Clear`. In fact, the loop that calls `_PyModule_Clear` has in its comments:
> Now, if there are any modules left alive, clear their globals to minimize potential leaks. All C extension modules actually end up here, since they are kept alive in the interpreter state.
That means that we can't clear the module state (which is used by C Extensions) before we run that loop.
Moving `_PyInterpreterState_ClearModules` until after it, fixes the segfault in the code snippet.
Finally, this updates a test in `io` to correctly assert the error that it now throws (since it now finds the io module state). The test that uses this is: `test_create_at_shutdown_without_encoding`. Given this test is now working is a proof that the module state now stays alive even when `__del__` is called at module destruction time. Thus, I didn't add a new tests for this.
https://bugs.python.org/issue38076
PyThreadState.on_delete is a callback used to notify Python when a
thread completes. _thread._set_sentinel() function creates a lock
which is released when the thread completes. It sets on_delete
callback to the internal release_sentinel() function. This lock is
known as Threading._tstate_lock in the threading module.
The release_sentinel() function uses the Python C API. The problem is
that on_delete is called late in the Python finalization, when the C
API is no longer fully working.
The PyThreadState_Clear() function now calls the
PyThreadState.on_delete callback. Previously, that happened in
PyThreadState_Delete().
The release_sentinel() function is now called when the C API is still
fully working.
Previously, a calltip might be left after SyntaxError, KeyboardInterrupt, or Shell Restart.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Tal Einat <taleinat+github@gmail.com>
Replace check for whether something is a method in the mock module. The
previous version fails on PyPy, because there no method wrappers exist
(everything looks like a regular Python-defined function). Thus the
isinstance(getattr(result, '__get__', None), MethodWrapperTypes) check
returns True for any descriptor, not just methods.
This condition could also return erroneously True in CPython for
C-defined descriptors.
Instead to decide whether something is a method, just check directly
whether it's a function defined on the class. This passes all tests on
CPython and fixes the bug on PyPy.
Some of the *SetItem methods in the C API steal a reference to the
given value. This annotates the better behaved ones to assure the
reader that these are not the ones with the inconsistent behaviour.
* 📜🤖 Added by blurb_it.
* make docs consistent with signature
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Improve multi-threaded performance by dropping the GIL in the fast path
of bytes.join. To avoid increasing overhead for small joins, it is only
done if the output size exceeds a threshold.
As Windows 7 is not supported by Python 3.9, we just replace the dynamic load with a static import. Backports will have a different fix to ensure they continue to behave the same.
In bpo-36264 os.path.expanduser was changed to ignore HOME on Windows.
Path.expanduser/home still honored HOME despite being documented as behaving the same
as os.path.expanduser. This makes them also ignore HOME so that both implementations
behave the same way again.
Whether or not overlap regions for self-intersecting polygons
or multiple shapes are filled depends on the operating system graphics,
typeof overlap, and number of overlaps.
Expose dialog buttons to test code and complete their test coverage.
Complete test coverage for highlights and keys tabs.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* Add DICT_UPDATE and DICT_MERGE bytecodes. Use them for ** unpacking.
* Remove BUILD_MAP_UNPACK and BUILD_MAP_UNPACK_WITH_CALL, as they are now unused.
* Update magic number for ** unpacking opcodes.
* Update dis.rst to incorporate new bytecodes.
* Add blurb entry.
The os.putenv() and os.unsetenv() functions are now always available.
On non-Windows platforms, Python now requires setenv() and unsetenv()
functions to build.
Remove putenv_dict from posixmodule.c: it's not longer needed.
As the function was not registering in the active patches, the mocks
started by `mock.patch.dict` were not being stopped when
`mock.patch.stopall` was being called.
* bpo-39421: Fix posible crash in heapq with custom comparison operators
* fixup! bpo-39421: Fix posible crash in heapq with custom comparison operators
* fixup! fixup! bpo-39421: Fix posible crash in heapq with custom comparison operators
* Add three new bytecodes: LIST_TO_TUPLE, LIST_EXTEND, SET_UPDATE. Use them to implement star unpacking expressions.
* Remove four bytecodes BUILD_LIST_UNPACK, BUILD_TUPLE_UNPACK, BUILD_SET_UNPACK and BUILD_TUPLE_UNPACK_WITH_CALL opcodes as they are now unused.
* Update magic number and dis.rst for new bytecodes.
* bpo-39336: Allow setattr to fail on modules which aren't assignable
When attaching a child module to a package if the object in sys.modules raises an AttributeError (e.g. because it is immutable) it causes the whole import to fail. This now allows immutable packages to exist and an ImportWarning is reported and the AttributeError exception is ignored.
When communicate() is called in a loop, it crashes when the child process
has already closed any piped standard stream, but still continues to be running
Co-authored-by: Andriy Maletsky <andriy.maletsky@gmail.com>
If setenv() C function is available, os.putenv() is now implemented
with setenv() instead of putenv(), so Python doesn't have to handle
the environment variable memory.
Deprecate binhex4 and hexbin4 standards. Deprecate the binhex module
and the following binascii functions:
* b2a_hqx(), a2b_hqx()
* rlecode_hqx(), rledecode_hqx()
* crc_hqx()
As described in RFC 1952, section 2.3.1, the XFL (eXtra FLags) byte of a
gzip member header should indicate whether the DEFLATE algorithm was
tuned for speed or compression ratio. Prior to this patch, archives
emitted by the `gzip` module always indicated maximum compression.
Fix comments and add tests for editor newline_and_indent_event method.
Remove unused None default for function parameter of pyparse find_good_parse_start method
and code triggered by that default.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
The public API symbols being removed are:
_PyBytes_InsertThousandsGroupingLocale, _PyBytes_InsertThousandsGrouping, _Py_InitializeFromArgs, _Py_InitializeFromWideArgs, _PyFloat_Repr, _PyFloat_Digits,
_PyFloat_DigitsInit, PyFrame_ExtendStack, _PyAIterWrapper_Type, PyNullImporter_Type, PyCmpWrapper_Type, PySortWrapper_Type, PyNoArgsFunction.
Remove the buffering parameter of bz2.BZ2File. Since Python 3.0, it
was ignored and using it was emitting a DeprecationWarning. Pass an
open file object to control how the file is opened.
The compresslevel parameter becomes keyword-only.
Remove base64.encodestring() and base64.decodestring(), aliases
deprecated since Python 3.1: use base64.encodebytes() and
base64.decodebytes() instead.
pstats is really useful or profiling and printing the output of the execution of some block of code, but I've found on multiple occasions when I'd like to access this output directly in an easily usable dictionary on which I can further analyze or manipulate.
The proposal is to add a function called get_profile_dict inside of pstats that'll automatically return this data the data in an easily accessible dict.
The output of the following script:
```
import cProfile, pstats
import pprint
from pstats import func_std_string, f8
def fib(n):
if n == 0:
return 0
if n == 1:
return 1
return fib(n-1) + fib(n-2)
pr = cProfile.Profile()
pr.enable()
fib(5)
pr.create_stats()
ps = pstats.Stats(pr).sort_stats('tottime', 'cumtime')
def get_profile_dict(self, keys_filter=None):
"""
Returns a dict where the key is a function name and the value is a dict
with the following keys:
- ncalls
- tottime
- percall_tottime
- cumtime
- percall_cumtime
- file_name
- line_number
keys_filter can be optionally set to limit the key-value pairs in the
retrieved dict.
"""
pstats_dict = {}
func_list = self.fcn_list[:] if self.fcn_list else list(self.stats.keys())
if not func_list:
return pstats_dict
pstats_dict["total_tt"] = float(f8(self.total_tt))
for func in func_list:
cc, nc, tt, ct, callers = self.stats[func]
file, line, func_name = func
ncalls = str(nc) if nc == cc else (str(nc) + '/' + str(cc))
tottime = float(f8(tt))
percall_tottime = -1 if nc == 0 else float(f8(tt/nc))
cumtime = float(f8(ct))
percall_cumtime = -1 if cc == 0 else float(f8(ct/cc))
func_dict = {
"ncalls": ncalls,
"tottime": tottime, # time spent in this function alone
"percall_tottime": percall_tottime,
"cumtime": cumtime, # time spent in the function plus all functions that this function called,
"percall_cumtime": percall_cumtime,
"file_name": file,
"line_number": line
}
func_dict_filtered = func_dict if not keys_filter else { key: func_dict[key] for key in keys_filter }
pstats_dict[func_name] = func_dict_filtered
return pstats_dict
pp = pprint.PrettyPrinter(depth=6)
pp.pprint(get_profile_dict(ps))
```
will produce:
```
{"<method 'disable' of '_lsprof.Profiler' objects>": {'cumtime': 0.0,
'file_name': '~',
'line_number': 0,
'ncalls': '1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'create_stats': {'cumtime': 0.0,
'file_name': '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/cProfile.py',
'line_number': 50,
'ncalls': '1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'fib': {'cumtime': 0.0,
'file_name': 'get_profile_dict.py',
'line_number': 5,
'ncalls': '15/1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'total_tt': 0.0}
```
As an example, this can be used to generate a stacked column chart using various visualization tools which will assist in easily identifying program bottlenecks.
https://bugs.python.org/issue37958
Automerge-Triggered-By: @gpshead
On Unix, subprocess.Popen.send_signal() now polls the process status.
Polling reduces the risk of sending a signal to the wrong process if
the process completed, the Popen.returncode attribute is still None,
and the pid has been reassigned (recycled) to a new different
process.
* Reorder the __aenter__ and __aexit__ checks for async with
* Add assertions for async with body being skipped
* Swap __aexit__ and __aenter__ loading in the documentation
Break up COMPARE_OP into four logically distinct opcodes:
* COMPARE_OP for rich comparisons
* IS_OP for 'is' and 'is not' tests
* CONTAINS_OP for 'in' and 'is not' tests
* JUMP_IF_NOT_EXC_MATCH for checking exceptions in 'try-except' statements.
This adds a new function named _PyErr_GetExcInfo() that is a variation of the
original PyErr_GetExcInfo() taking a PyThreadState as its first argument.
That function allows to retrieve the exceptions information of any Python
thread -- not only the current one.
Copying property objects results in a TypeError. Steps to reproduce:
```
>>> import copy
>>> obj = property()
>>> copy.copy(obj)
````
This affects both shallow and deep copying.
My idea for a fix is to add property objects to the list of "atomic" objects in the copy module.
These already include types like functions and type objects.
I also added property objects to the unit tests test_copy_atomic and test_deepcopy_atomic. This is my first PR, and it's highly likely I've made some mistake, so please be kind :)
https://bugs.python.org/issue38293
nntplib.NNTP and nntplib.NNTP_SSL now raise a ValueError
if the given timeout for their constructor is zero to
prevent the creation of a non-blocking socket.
poplib.POP3 and poplib.POP3_SSL now raise a ValueError
if the given timeout for their constructor is zero to
prevent the creation of a non-blocking socket.
The fix changes copy_location() to require an extra node from which to extract the end location, and fixing all 5 call sites.
https://bugs.python.org/issue39235
imaplib.IMAP4 and imaplib.IMAP4_SSL now have an
optional *timeout* parameter for their constructors.
Also, the imaplib.IMAP4.open() method now has an optional *timeout* parameter
with this change. The overridden methods of imaplib.IMAP4_SSL and
imaplib.IMAP4_stream were applied to this change.
When producing the bytecode of exception handlers with name binding (like `except Exception as e`) we need to produce a try-finally block to make sure that the name is deleted after the handler is executed to prevent cycles in the stack frame objects. The bytecode associated with this try-finally block does not have source lines associated and it was causing problems when the tracing functionality was running over it.
This uses the heuristic of assuming a named tuple is a subclass of
tuple with a _fields attribute. This change means that contents of
a named tuple wouldn't be converted - if a user wants to have
ConvertingTuple functionality from a namedtuple, they will have to
implement it themselves.
The importlib.metadata documentation uses hardcoded links to internal
pages. This results in minor rendering issues. This change replaces
the hardcoded links with suitable Sphinx roles.
Signed-off-by: Oleg Höfling <oleg.hoefling@gmail.com>
Small typo/formatting corrections.
`whethen` -> `whether`
`exaustion' -> `exhaustion`
Assorted appending periods `.` and slight reformattings to place `Path contributed by` on the same line as description, matching the majority of document.
NB Some of these might need to be backported, as I saw the first error in the [changelog for 3.8.1](https://docs.python.org/3.8/whatsnew/changelog.html#python-3-8-1)
When checking `setup.py` and when the `author` field was provided, but
the `author_email` field was missing, erroneously a warning message was
displayed that the `author_email` field is required.
The specs do not require the `author_email`field:
https://packaging.python.org/specifications/core-metadata/#author
The same is valid for `maintainer` and `maintainer_email`.
The warning message has been adjusted.
modified: Doc/distutils/examples.rst
modified: Lib/distutils/command/check.py
https://bugs.python.org/issue38914
Fix test_ressources_gced_in_workers() of test_concurrent_futures:
explicitly stop the manager to prevent leaking a child process
running in the background after the test completes.
All keywords should first be checked for pointer identity. Only
after that failed for all keywords (unlikely) should unicode
equality be used.
The original code would call unicode equality on any non-matching
keyword argument. Meaning calling it often e.g. when a function
has many kwargs but only the last one is provided.
Multiprocessing and concurrent.futures tests now stop the resource
tracker process when tests complete.
Add ResourceTracker._stop() method to
multiprocessing.resource_tracker.
Add _cleanup_tests() helper function to multiprocessing.util: share
code between multiprocessing and concurrent.futures tests.
Each Python subinterpreter now has its own "small integer
singletons": numbers in [-5; 257] range.
It is no longer possible to change the number of small integers at
build time by overriding NSMALLNEGINTS and NSMALLPOSINTS macros:
macros should now be modified manually in pycore_pystate.h header
file.
For now, continue to share _PyLong_Zero and _PyLong_One singletons
between all subinterpreters.
Commit 6b5b013bcc ("bpo-26978: Implement pathlib.Path.link_to (Using
os.link) (GH-12990)") introduced a new link_to method in pathlib. However,
this makes pathlib crash when the 'os' module is missing a 'link' method.
Fix this by checking for the presence of the 'link' method on pathlib
module import, and if it's not present, turn it into a runtime error like
those emitted when there is no lchmod() or symlink().
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Provides a richer platform tag for AIX that we expect to be sufficient for PEP 425
binary distribution identification. Any backports to earlier Python versions will be
handled via setuptools.
Patch by Michael Felt.
When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node.
https://bugs.python.org/issue39031
Automerge-Triggered-By: @pablogsal
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.
SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.
The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.
Uniformize also "from test import support" import in some test files.
* bpo-39022, bpo-38594: Sync with importlib_metadata 1.3 including improved docs for custom finders and better serialization support in EntryPoints.
* 📜🤖 Added by blurb_it.
* Correct module reference
In Python 3.9.0a1, sys.argv[0] was made an asolute path if a filename
was specified on the command line. Revert this change, since most
users expect sys.argv to be unmodified.
Fix test_pty: if the process is the session leader, closing the
master file descriptor raises a SIGHUP signal: simply ignore SIGHUP
when running the tests.
test_openssl_version now accepts version 3.0.0.
getpeercert() no longer returns IPv6 addresses with a trailing new line.
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38820
On most platforms, the `environ` symbol is accessible everywhere.
In a dylib on OSX, it's not easily accessible, you need to find it with
_NSGetEnviron.
The code was caching the *value* of environ. But a setenv() can change the value,
leaving garbage at the old value. Fix: don't cache the value of environ, just
read it every time.
If waitpid() is called elsewhere, waitpid() call fails with
ChildProcessError: use return code 255 in this case, and log a
warning. It ensure that the pidfd file descriptor is closed if this
error occurs.
Break cycle generated when saving an exception in socket.py, codeop.py and dyld.py as they keep alive not only the exception but user objects through the ``__traceback__`` attribute.
https://bugs.python.org/issue36820
Automerge-Triggered-By: @pablogsal
parse_message_id() was improperly using a token defined inside an exception
handler, which was raising `UnboundLocalError` on parsing an invalid value.
https://bugs.python.org/issue38698
The readline module now detects if Python is linked to libedit at runtime
on all platforms. Previously, the check was only done on macOS.
If Python is used as a library by a binary linking to libedit, the linker
resolves the rl_initialize symbol required by the readline module against
libedit instead of libreadline, which leads to a segfault.
Take advantage of the existing supporting code to have readline module being
compatible with both situations.
Serhiy Storchaka
Shantanu
Victor Stinner
Zackery Spytz
Jakub Stasiak
Anthony Shaw
Saiyang Gou
Philipp Gesang
Stefan Behnel
Eddie Elizondo
Steve Cirelli
Pierre Glaser
Kyle Stanley
Andrew Svetlov
Dong-hee Na
Carl Friedrich Bolz-Tereick
Joannah Nanjekye
Bruce Merry
Steve Dower
David Carlier
Rémi Lapeyre
Adam Meily
Christoph Reiter
Terry Jan Reedy
Cheryl Sabella
Mark Shannon
Karthikeyan Singaravelan
加和
Vegard Stikbakke
Łukasz Langa
Matthew Kokotovich
mbarkhau
Mario Corchero
Pablo Galindo
William Woodruff
Dino Viehland
Alex Rebert
William Chargin
Inada Naoki
Daniel Olshansky
Hai Shi
Vinay Sajip
Václav Bartoš
Géry Ogam
Julien Danjou
Philip McMahon
Batuhan Taşkaya
Guðni Natan Gunnarsson
Jason R. Coombs
Guido van Rossum
An Long
Derek Brown
Chandan Singh
Anthony Sottile
Raymond Hettinger
Jendrik Seipp
Oleg Höfling
Gurupad Hegde
Aurora Lanes
toonarmycaptain
Éric Araujo
Jürgen Gmach
Bar Harel
Parth Sharma
Sebastian Berg
Lysandros Nikolaou
Toke Høiland-Jørgensen
Michael Felt
JohnnyNajera
Tim Gates
Yury Selivanov
Mark Dickinson
Abhilash Raj
Elena Oat
AMIR
xdegaye
Christian Heimes
Anj-A
idomic
Benoit Hudson
Mario Corchero
wim glenn
Claudiu Popa
serge-sans-paille
Daniel Himmelstein
Matthew Rollings
Daniel Hillier
Tzu-ping Chung
Bruno P. Kinoshita
HongWeipeng