Commit Graph

26945 Commits

Author SHA1 Message Date
Carl Meyer 82ccbf69a8
gh-91051: allow setting a callback hook on PyType_Modified (GH-97875) 2022-10-21 14:41:51 +01:00
Gregory P. Smith 49f61068f4
gh-97514: Don't use Linux abstract sockets for multiprocessing (#98501)
Linux abstract sockets are insecure as they lack any form of filesystem
permissions so their use allows anyone on the system to inject code into
the process.

This removes the default preference for abstract sockets in
multiprocessing introduced in Python 3.9+ via
https://github.com/python/cpython/pull/18866 while fixing
https://github.com/python/cpython/issues/84031.

Explicit use of an abstract socket by a user now generates a
RuntimeWarning.  If we choose to keep this warning, it should be
backported to the 3.7 and 3.8 branches.
2022-10-20 15:30:09 -07:00
Dennis Sweeney 39bc70e267
gh-97912: Avoid quadratic behavior when adding LOAD_FAST_CHECK (GH-97952)
* The compiler analyzes the usage of the first 64 local variables all at once using bit masks.

* Local variables beyond the first 64 are only partially analyzed, achieving linear time.
2022-10-20 18:27:41 -04:00
Ben Kallus 6f15ca8c7a
gh-96035: Make urllib.parse.urlparse reject non-numeric ports (#98273)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-10-20 14:00:56 -07:00
Irit Katriel 4ec9ed8fde
gh-98461: Fix source location in comprehensions bytecode (GH-98464) 2022-10-20 16:58:37 +01:00
Steve Dower e48f9b2b7e
gh-98360: multiprocessing now spawns children on Windows with correct argv[0] in virtual environments (GH-98462) 2022-10-20 14:53:38 +01:00
Noam Cohen a371a7e03e
gh-95023: Added os.setns and os.unshare functions (#95046)
Added os.setns and os.unshare to easily switch between namespaces
on Linux.

Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Victor Stinner <vstinner@python.org>
2022-10-20 11:08:54 +02:00
Dong-hee Na 4156b2fc13
gh-98374: Suppress ImportError for invalid query for help() command. (gh-98450) 2022-10-20 10:56:21 +09:00
chgnrdv 1f369ad07f
gh-98354: Add unicode check for 'name' attribute in _imp_create_builtin (GH-98412)
Fixes #98354
2022-10-19 17:25:10 -07:00
Victor Stinner a8fe4bbd6b
gh-98257: Make _PyEval_SetTrace() reentrant (#98258)
Make sys.setprofile() and sys.settrace() functions reentrant.  They
can no long fail with: RuntimeError("Cannot install a trace function
while another trace function is being installed").

Make _PyEval_SetTrace() and _PyEval_SetProfile() functions reentrant,
rather than detecting and rejecting reentrant calls. Only delete the
reference to function arguments once the new function is fully set,
when a reentrant call is safe. Call also _PySys_Audit() earlier.
2022-10-20 00:31:47 +02:00
Steve Dower 4bd63f66cd
gh-98414: py.exe launcher does not use defaults for -V:company/ option (GH-98460) 2022-10-19 23:00:09 +01:00
Hugo van Kemenade 50553004fe
Docs: Bump sphinx-lint and fix unbalanced inline literal markup (#98441)
Bump sphinx-lint and fix unbalanced inline literal markup
2022-10-19 14:00:28 +02:00
Jack Hindmarch b6e59d76c0
gh-92886: Replace assertion statements in `handlers.BaseHandler` to support running with optimizations (`-O`) (GH-93231) 2022-10-19 11:38:59 +01:00
Jack Hindmarch a3be874334
gh-92886: Fix tests that fail when running with optimizations (`-O`) in `_test_multiprocessing.py` (GH-93233) 2022-10-19 11:07:07 +01:00
Jack Hindmarch 602ea40d89
gh-92886: Fix tests that fail when running with optimizations (`-O`) in `test_py_compile.py` (GH-93235) 2022-10-19 11:05:08 +01:00
Irit Katriel 9be05df399
gh-98398: Fix source locations for 'assert' bytecode (GH-98405) 2022-10-19 10:48:50 +01:00
Serhiy Storchaka e4ec8de6fa
gh-97928: Fix handling options starting with "-" in tkinter.Text.count() (GH-98436)
Previously they were silently ignored. Now they are errors.
2022-10-19 12:30:14 +03:00
Irit Katriel c051d55ddb
gh-98390: Fix source locations of boolean sub-expressions (GH-98396) 2022-10-18 17:18:38 +01:00
Jack Hindmarch debacd9ad5
gh-92886: Fixing tests that fail when running with optimizations (`-O`) in `test_sys_settrace.py` (GH-93234) 2022-10-18 16:59:05 +01:00
Victor Stinner db03c8066a
gh-98393: os module reject bytes-like, only accept bytes (#98394)
The os module and the PyUnicode_FSDecoder() function no longer accept
bytes-like paths, like bytearray and memoryview types: only the exact
bytes type is accepted for bytes strings.
2022-10-18 17:52:31 +02:00
Paul Moore 9da5215000
gh-98331: Update bundled pip to 22.3 (#98332) 2022-10-18 15:48:14 +01:00
Raymond Hettinger de3ece769a
GH-98363: Add itertools.batched() (GH-98364) 2022-10-17 18:53:45 -05:00
Victor Stinner 1863302d61
gh-97669: Create Tools/build/ directory (#97963)
Create Tools/build/ directory. Move the following scripts from
Tools/scripts/ to Tools/build/:

* check_extension_modules.py
* deepfreeze.py
* freeze_modules.py
* generate_global_objects.py
* generate_levenshtein_examples.py
* generate_opcode_h.py
* generate_re_casefix.py
* generate_sre_constants.py
* generate_stdlib_module_names.py
* generate_token.py
* parse_html5_entities.py
* smelly.py
* stable_abi.py
* umarshal.py
* update_file.py
* verify_ensurepip_wheels.py

Update references to these scripts.
2022-10-17 12:01:00 +02:00
Ruben Vorderman eae7dad402
gh-95534: Improve gzip reading speed by 10% (#97664)
Change summary:
+ There is now a `gzip.READ_BUFFER_SIZE` constant that is 128KB. Other programs that read in 128KB chunks: pigz and cat. So this seems best practice among good programs. Also it is faster than 8 kb chunks.
+ a zlib._ZlibDecompressor was added. This is the _bz2.BZ2Decompressor ported to zlib. Since the zlib.Decompress object is better for in-memory decompression, the _ZlibDecompressor is hidden. It only makes sense in file decompression, and that is already implemented now in the gzip library. No need to bother the users with this.
+ The ZlibDecompressor uses the older Cpython arrange_output_buffer functions, as those are faster and more appropriate for the use case. 
+ GzipFile.read has been optimized. There is no longer a `unconsumed_tail` member to write back to padded file. This is instead handled by the ZlibDecompressor itself, which has an internal buffer. `_add_read_data` has been inlined, as it was just two calls.

EDIT: While I am adding improvements anyway, I figured I could add another one-liner optimization now to the python -m gzip application. That read chunks in io.DEFAULT_BUFFER_SIZE previously, but has been updated now to use READ_BUFFER_SIZE chunks.
2022-10-16 19:10:58 -07:00
Jason R. Coombs cea910ebf1
gh-97930: Merge with importlib_resources 5.9 (GH-97929)
* Merge with importlib_resources 5.9

* Update changelog
2022-10-16 15:00:39 -04:00
Joannah Nanjekye 5c9302d03a
gh-85525: Remove extra row in doc (#98337)
* remove extra row

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-10-16 13:43:31 -04:00
Terry Jan Reedy 35fa5d5e7f
gh-97527: IDLE - fix buggy macosx patch (#98313)
#97530 fixed IDLE tests possibly crashing on a Mac without a GUI.
But it resulted in IDLE not starting in 3.10.8, 3.12.0a1, and
Microsoft Python 3.10.2288.0 when test/* is not installed.
After this patch, test.* is only imported when testing on Mac.
2022-10-16 10:23:11 -04:00
Vinay Sajip f6b1e4048d
gh-98307: Add docstring and documentation for SysLogHandler.createSocket (GH-98319)
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-10-16 09:15:46 +01:00
Kumar Aditya 660f10248b
GH-94597: Deprecate child watcher getters and setters (#98215)
This is the next step for deprecating child watchers.

Until we've removed the API completely we have to use it, so this PR is mostly suppressing a lot of warnings when using the API internally.

Once the child watcher API is totally removed, the two child watcher implementations we actually use and need (Pidfd and Thread) will be turned into internal helpers.
2022-10-15 16:09:30 -07:00
Pablo Galindo Salgado bb56dead33
gh-98254: Include stdlib module names in error messages for NameErrors (#98255) 2022-10-15 23:13:33 +01:00
Jakub Kuczys 120b4ab2b6
gh-95731: Fix module docstring extraction in pygettext (#95732) 2022-10-15 07:57:53 -07:00
Steve Dower 2fe44f728a
gh-98251: Allow venv to pass along PYTHON* variables to pip and ensurepip when they do not impact path resolution (GH-98259) 2022-10-14 16:58:54 +01:00
Tony Solomonik b863b9cd4b
Bpo-41246: IOCP Proactor avoid callback code duplication (#21399)
Use the same callback function for overlapped operations recv, recv_into, recvfrom, sendto, send, and sendfile inside IocpProactor.
2022-10-13 11:05:16 -07:00
Christoph Hamsen c9ed0327a9
bpo-46364: Use sockets for stdin of asyncio only on AIX (#30596)
Signed-off-by: Christoph Hamsen <hamsen.christoph@posteo.de>
Co-authored-by: July Tikhonov <july.tikh@gmail.com>
2022-10-13 09:11:15 -07:00
Victor Stinner d4b9166385
gh-98178: syslog() is not thread-safe on macOS (#98213)
On macOS, fix a crash in syslog.syslog() in multi-threaded
applications. On macOS, the libc syslog() function is not
thread-safe, so syslog.syslog() no longer releases the GIL to call
it.
2022-10-13 13:34:55 +02:00
Julien Palard 4067c6d7fe
gh-86404: Doc: Drop now unused make suspicious and rstlint. (GH-98179)
They have been replaced by
[sphinx-lint](https://github.com/sphinx-contrib/sphinx-lint).
2022-10-11 15:31:33 +02:00
Ken Jin b399115ef1
gh-95756: Lazily created cached co_* attrs (GH-97791) 2022-10-11 11:26:08 +08:00
Shantanu f871e9a7bb
gh-44098: Release the GIL during mmap on Unix (GH-98146)
This seems pretty straightforward. The issue mentions other calls in mmapmodule that we could release the GIL on, but those are in methods where we'd need to be careful to ensure that something sensible happens if those are called concurrently. In prior art, note that #12073 released the GIL for munmap.  In a toy benchmark, I see the speedup you'd expect from doing this.

Automerge-Triggered-By: GH:gvanrossum
2022-10-10 15:14:31 -07:00
Matthias Görgens 553d3c1017
gh-96821: Fix undefined behaviour in `audioop.c` (#96923)
* gh-96821: Fix undefined behaviour in `audioop.c`

Left-shifting negative numbers is undefined behaviour.

Fortunately, multiplication works just as well, is defined behaviour,
and gets compiled to the same machine code as before by optimizing
compilers.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-10-10 17:12:29 +01:00
Carl Meyer ad817cd5c4
bpo-43564: preserve original exception in args of FTP URLError (#24938)
* bpo-43564: preserve original error in args of FTP URLError

* Add NEWS blurb

Co-authored-by: Carl Meyer <carljm@instagram.com>
2022-10-09 18:59:07 -07:00
Kumar Aditya d8765284f3
GH-94597: deprecate `SafeChildWatcher`, `FastChildWatcher` and `MultiLoopChildWatcher` child watchers (#98089) 2022-10-08 13:52:19 -07:00
Joannah Nanjekye 4ed00be98f
gh-68686: Retire eptag ptag scripts (#98064)
* Retire eptag ptag scripts

* 📜🤖 Added by blurb_it.

* fix news entry error

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-10-08 07:57:47 -07:00
Pablo Galindo Salgado 83eb827247
gh-97922: Run the GC only on eval breaker (#97920) 2022-10-08 07:57:09 -07:00
Kumar Aditya 8ba9378b16
GH-98023: Change default child watcher to PidfdChildWatcher on supported systems (#98024) 2022-10-07 17:29:09 -07:00
Thomas Grainger 3d8b224547
GH-94182: Run the PidfdChildWatcher on the running loop (#94184)
There is no reason for this watcher to be attached to any particular loop.
This should make it safe to use regardless of the lifetime of the event loop running in the main thread
(relative to other loops).

Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-10-07 17:24:01 -07:00
Shantanu cb4615fd43
gh-90085: Remove vestigial -t and -c timeit options (#94941)
See bpo-28240. The functionality was removed in 3d7feb9ac2.
The options had been deprecated since Python 3.3
2022-10-07 15:38:20 -07:00
Frazer McLean f4f8133387
GH-83901: Improve Signature.bind error message for missing keyword-only params (#95347)
Fixes GH-83901
2022-10-07 15:24:17 -07:00
Jelle Zijlstra 0f111f53c1
gh-96073: fix backticks in NEWS entry (GH-98056)
Automerge-Triggered-By: GH:JelleZijlstra
2022-10-07 14:44:56 -07:00
Lysandros Nikolaou 3de08ce8c1
gh-97997: Add col_offset field to tokenizer and use that for AST nodes (#98000) 2022-10-07 14:38:35 -07:00
Serhiy Storchaka c11b667a1d
gh-96346: Use double caching for re._compile() (#96347) 2022-10-07 12:21:42 -07:00
Noam Cohen 2a168355f0
gh-97646: Change `.js` and `.mjs` files mimetype to conform to RFC 9239 (#97934) 2022-10-07 12:00:53 -07:00
Anh71me d5fea01d9d
GH-96073: Fix wild replacement in inspect.formatannotation (#96074)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-10-07 11:23:06 -07:00
Nikita Sobolev 24a6645894
gh-97955: Migrate `zoneinfo` to Argument Clinic (#97958) 2022-10-07 11:06:23 -07:00
Nikita Sobolev 83cbe84dc2
gh-64373: Convert `_functools` to Argument Clinic (#96640) 2022-10-07 10:36:40 -07:00
Nikita Sobolev 5ba4875aec
gh-96415: Remove `types._cell_factory` from a module namespace (#96416)
Closes #96415
2022-10-07 10:27:14 -07:00
Barry Warsaw 13d4489142
gh-86298: Ensure that __loader__ and __spec__.loader agree in warnings.warn_explicit() (GH-97803)
In `_warnings.c`, in the C equivalent of `warnings.warn_explicit()`, if the module globals are given (and not None), the warning will attempt to get the source line for the issued warning.  To do this, it needs the module's loader.

Previously, it would only look up `__loader__` in the module globals.  In https://github.com/python/cpython/issues/86298 we want to defer to the `__spec__.loader` if available.

The first step on this journey is to check that `loader == __spec__.loader` and issue another warning if it is not.  This commit does that.

Since this is a PoC, only manual testing for now.

```python
# /tmp/foo.py
import warnings

import bar

warnings.warn_explicit(
    'warning!',
    RuntimeWarning,
    'bar.py', 2,
    module='bar knee',
    module_globals=bar.__dict__,
    )
```

```python
# /tmp/bar.py
import sys
import os
import pathlib

# __loader__ = pathlib.Path()
```

Then running this: `./python.exe -Wdefault /tmp/foo.py`

Produces:

```
bar.py:2: RuntimeWarning: warning!
  import os
```

Uncomment the `__loader__ = ` line in `bar.py` and try it again:

```
sys:1: ImportWarning: Module bar; __loader__ != __spec__.loader (<_frozen_importlib_external.SourceFileLoader object at 0x109f7dfa0> != PosixPath('.'))
bar.py:2: RuntimeWarning: warning!
  import os
```

Automerge-Triggered-By: GH:warsaw
2022-10-06 19:32:53 -07:00
Gregory P. Smith 27369ef56f
gh-82874: Convert remaining importlib format uses to f-str. (#98005)
f-yes
2022-10-06 18:27:51 -07:00
Hugo van Kemenade fa2d43e518
Docs: Fix backtick errors found by sphinx-lint (#97998)
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
2022-10-06 18:01:30 -07:00
Nikita Sobolev 27025e158c
gh-97850: Remove deprecated functions from `importlib.utils` (#97898)
* gh-97850: Remove deprecated functions from `importlib.utils`

* Rebase and remove `set_package` from diff
2022-10-06 17:57:10 -07:00
Guido van Rossum 09de8d7aaf
GH-90985: Revert "Deprecate passing a message into cancel()" (#97999)
Reason: we were too hasty in deprecating this.
We shouldn't deprecate it before we have a replacement.
2022-10-06 17:30:27 -07:00
Tiger c46a423a52
bpo-35540 dataclasses.asdict now supports defaultdict fields (gh-32056) 2022-10-06 17:11:59 -07:00
Carl Meyer a4b7794887
GH-91052: Add C API for watching dictionaries (GH-31787) 2022-10-07 01:08:00 +01:00
Gregory P. Smith 683ab85955
bpo-38693: Use f-strings instead of str.format() within importlib (#17058)
This is a small performance improvement, especially for one or two hot
places such as _handle_fromlist() that are called a lot and the
.format() method was being used just to join two strings with a dot.

Otherwise it is merely a readability improvement.

We keep `_ERR_MSG` and `_ERR_MSG_PREFIX` as those may be used elsewhere for canonical looking error messages.
2022-10-06 16:43:16 -07:00
Brandt Bucher 21a2d9ff55
GH-97002: Prevent `_PyInterpreterFrame`s from backing more than one `PyFrameObject` (GH-97996) 2022-10-07 00:20:01 +01:00
Lysandros Nikolaou cbf0afd8a1
gh-97973: Return all necessary information from the tokenizer (GH-97984)
Right now, the tokenizer only returns type and two pointers to the start and end of the token.
This PR modifies the tokenizer to return the type and set all of the necessary information,
so that the parser does not have to this.
2022-10-06 16:07:17 -07:00
Dong-hee Na b9d2e81716
fixes gh-96078: os.sched_yield release the GIL while calling sched_yield(2). (gh-97965) 2022-10-06 15:57:37 -07:00
Brett Cannon e1c4d56fdd
gh-65961: Do not rely solely on `__cached__` (GH-97990)
Make sure `__spec__.cached` (at minimum) can be used.
2022-10-06 15:40:22 -07:00
Jason R. Coombs 8af04cdef2
gh-97781: Apply changes from importlib_metadata 5. (GH-97785)
* gh-97781: Apply changes from importlib_metadata 5.

* Apply changes from upstream

* Apply changes from upstream.
2022-10-06 15:25:24 -04:00
Terry Jan Reedy 2b5f1360ea
gh-86482: Document assignment expression need for ()s (#23291)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-10-06 12:24:17 -07:00
larryhastings 6bfb0be804
gh-97943: PyFunction_GetAnnotations should return a borrowed reference. (#97949) 2022-10-06 12:23:20 -07:00
Ned Deily 6d0a0191a4
gh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (GH-97944)
The macOS 13 SDK includes support for the `mkfifoat` and `mknodat` system calls.
 Using the `dir_fd` option with either `os.mkfifo` or `os.mknod` could result in a
 segfault if cpython is built with the macOS 13 SDK but run on an earlier
 version of macOS. Prevent this by adding runtime support for detection of
 these system calls ("weaklinking") as is done for other newer syscalls on
 macOS.
2022-10-06 09:11:47 -07:00
Ethan Furman b44372e03c
gh-96865: [Enum] fix Flag to use CONFORM boundary (GH-97528) 2022-10-05 15:25:55 -07:00
Brett Cannon c206e53bb7
gh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (#97879)
Also remove `importlib.util.set_package()` which was already slated for removal.

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2022-10-05 15:00:45 -07:00
Barry Warsaw 5dc3599135
gh-97850: Remove all known instances of module_repr() (#97876)
Remove all known instances of module_repr()
2022-10-05 11:42:26 -07:00
Tshepang Mbambo 815008a3a5
I changed my surname early this year (#96671)
* I recently changed my name

* Update ACKS
2022-10-05 11:16:45 -07:00
Adam Turner 0031e62973
gh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (#97768)
:c:type:`<C type>` -> :c:expr:`<C type>`

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2022-10-05 11:01:14 -07:00
Pieter Eendebak aeb28f5130
gh-91539: improve performance of get_proxies_environment (#91566)
* improve performance of get_proxies_environment when there are many environment variables

* 📜🤖 Added by blurb_it.

* fix case of short env name

* fix formatting

* fix whitespace

* whitespace

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <carl@oddbird.net>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <carl@oddbird.net>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <carl@oddbird.net>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <carl@oddbird.net>

* whitespace

* Update Misc/NEWS.d/next/Library/2022-04-15-11-29-38.gh-issue-91539.7WgVuA.rst

Co-authored-by: Carl Meyer <carl@oddbird.net>

* Update Lib/urllib/request.py

Co-authored-by: Carl Meyer <carl@oddbird.net>

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Carl Meyer <carl@oddbird.net>
2022-10-05 10:57:52 -07:00
Kumar Aditya 7015e13797
gh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (#32073) 2022-10-05 10:15:31 -07:00
Serhiy Storchaka 77f0249308
gh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (GH-96383) 2022-10-05 14:21:16 +03:00
Serhiy Storchaka e3ef400be7
gh-74696: Pass root_dir to custom archivers which support it (GH-94251)
Co-authored-by: Éric <merwok@netwok.org>
2022-10-05 12:48:59 +03:00
Guido van Rossum 8079bef56f
GH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (#96756)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-10-04 23:49:10 -07:00
andrei kulakov db64fb9bbe
gh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (#97826)
* fix AttributeError, add unit test
2022-10-04 17:47:49 -07:00
Mark Shannon 76449350b3
GH-91079: Decouple C stack overflow checks from Python recursion checks. (GH-96510) 2022-10-05 01:34:03 +01:00
Brandt Bucher 0ff8fd6583
GH-97779: Ensure that *all* frame objects are backed by "complete" frames (GH-97845) 2022-10-04 17:30:03 -07:00
Nikita Sobolev c3648f4e4a
gh-97837: Change deprecation warning message in `unittest` (#97838) 2022-10-04 17:29:18 -07:00
Adam Turner 6b3d4db02e
gh-93738: Documentation C syntax (:c:type: to :c:expr:, misc. cases) (#97775)
* :c:type: to :c:expr:

* Update Doc/whatsnew/2.4.rst
2022-10-04 16:32:27 -07:00
Ev2geny 743453a554
gh-58451: Add optional delete_on_close parameter to NamedTemporaryFile (GH-97015) 2022-10-04 23:37:33 +01:00
Łukasz Langa bbc7cd649a
gh-97008: Add a Python implementation of AttributeError and NameError suggestions (#97022)
Relevant tests moved from test_exceptions to test_traceback to be able to
compare both implementations.

Co-authored-by: Carl Friedrich Bolz-Tereick <cfbolz@gmx.de>
2022-10-04 15:31:16 -07:00
Nikita Sobolev 4f380db1a5
gh-96142: add missing params to `dataclass._DataclassParams` (gh-96382) 2022-10-04 09:53:28 -07:00
Victor Stinner 116fa62c6e
gh-97670: Remove sys.getdxp() and analyze_dxp.py script (#97671)
Remove the sys.getdxp() function and the Tools/scripts/analyze_dxp.py
script. DXP stands for "dynamic execution pairs". They were related
to DYNAMIC_EXECUTION_PROFILE and DXPAIRS macros which have been
removed in Python 3.11. Python can now be built with "./configure
--enable-pystats" to gather statistics on Python opcodes.
2022-10-04 15:28:57 +02:00
William Woodruff 6e53308829
Adjust stable ABI internal documentation (GH-96896)
I was perusing this file, and noticed that this part of the documentation is slightly out of date: the `struct` items in this TOML file currently contain `struct_abi_kind` members, which distinguish between the different types of ABI compatibility described in the comment.

I've updated the comment to reflect this.
2022-10-04 06:05:53 -07:00
Victor Stinner 6cbbc26a73
gh-97669: Remove outdated example scripts (#97675)
Remove outdated example scripts of the Tools/scripts/ directory. A
copy can be found in the old-demos project:
https://github.com/gvanrossum/old-demos

Removed scripts (39):

* byext.py
* byteyears.py
* cleanfuture.py
* copytime.py
* crlf.py
* db2pickle.py
* dutree.doc
* dutree.py
* find-uname.py
* find_recursionlimit.py
* finddiv.py
* findlinksto.py
* findnocoding.py
* fixcid.py
* fixdiv.py
* fixheader.py
* fixnotice.py
* fixps.py
* get-remote-certificate.py
* google.py
* highlight.py
* ifdef.py
* import_diagnostics.py
* lfcr.py
* linktree.py
* lll.py
* mailerdaemon.py
* make_ctype.py
* mkreal.py
* objgraph.py
* pdeps.py
* pickle2db.py
* pindent.py
* pysource.py
* reindent-rst.py
* rgrep.py
* suff.py
* texi2html.py
* which.py

Changes:

* Remove test_fixcid, test_lll, test_pdeps and test_pindent
  of test.test_tools.
* Remove get-remote-certificate.py changelog entry, since the script
  was removed.

Note: there is a copy of crlf.py in Lib/test/test_lib2to3/data/.
2022-10-04 10:49:00 +02:00
Koki Saito 27e59afa2a
gh-97816: Remove unused variables in `mutliprocessing.managers.Server` (#97817)
Remove unused local variables.
2022-10-03 22:29:17 -07:00
Nikita Sobolev 06016845dc
gh-97639: Remove `tokenize.NL` check from `tabnanny` (#97640)
* gh-97639: Remove `tokenize.NL` check from `tabnanny`

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-10-03 17:41:29 -07:00
Brandt Bucher 93fcc1f413
GH-97752: Clear the `previous` member of newly-created generator/coroutine frames (GH-97795) 2022-10-04 00:36:52 +01:00
larryhastings 00b5a08c80
gh-97799: use inspect.get_annotations in dataclass (#97800)
dataclass used to get the annotations on a class object using
cls.__dict__.get('__annotations__').  Now that it always imports
inspect, it can use inspect.get_annotations, which is modern
best practice for coping with annotations.
2022-10-03 15:46:09 -07:00
Gregory P. Smith b0f89cb431
gh-96512: Move int_max_str_digits setting to PyConfig (#96944)
It had to live as a global outside of PyConfig for stable ABI reasons in
the pre-3.12 backports.

This removes the `_Py_global_config_int_max_str_digits` and gets rid of
the equivalent field in the internal `struct _is PyInterpreterState` as
code can just use the existing nested config struct within that.

Adds tests to verify unique settings and configs in subinterpreters.
2022-10-03 13:55:45 -07:00
Victor Stinner 64fe343717
gh-97681: Remove Tools/demo/ directory (#97682)
Remove the Tools/demo/ directory which contained old demo scripts. A
copy can be found in the old-demos project:
https://github.com/gvanrossum/old-demos

Remove the following old demo scripts:

* beer.py
* eiffel.py
* hanoi.py
* life.py
* markov.py
* mcast.py
* queens.py
* redemo.py
* rpython.py
* rpythond.py
* sortvisu.py
* spreadsheet.py
* vector.py

Changes:

* Remove a reference to the redemo.py script in the regex howto
  documentation.
* Remove a reference to the removed Tools/demo/ directory in the
  curses documentation.
* Update PC/layout/ to remove the reference to Tools/demo/ directory.
2022-10-03 17:09:02 +02:00
Serhiy Storchaka 0ee9619a4c
gh-97728: Argument Clinic: Fix uninitialized variable in the Py_UNICODE converter (GH-97729)
It affects function os.system() on Windows and Windows-specific modules
winreg, _winapi, _overlapped, and _msi.
2022-10-03 10:42:54 +03:00
C.A.M. Gerlach 9148c0d893
gh-97740: Fix bang in Sphinx C domain ref target syntax (#97741)
* gh-97740: Fix bang in Sphinx C domain ref target syntax

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

* Add NEWS entry for C domain bang fix

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2022-10-03 05:51:22 +02:00
Koki Saito 19ca114645
gh-96819: multiprocessing.resource_tracker: check if length of pipe write <= 512 (#96890)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-10-02 17:41:01 -07:00
Ofey Chan d639438609
gh-97591: In `Exception.__setstate__()` acquire strong references before calling `tp_hash` slot (#97700) 2022-10-01 20:57:17 -07:00
Gregory P. Smith 8baef8ae36
gh-95588: Drop the safety claim from `ast.literal_eval` docs. (#95919)
It was never really safe and this claim conflicts directly with the big warning in the docs about it being able to crash the interpreter.
2022-10-01 17:55:40 -07:00
Guido van Rossum 63780f4599
GH-97592: Fix crash in C remove_done_callback due to evil code (#97660)
Evil code could cause fut_callbacks to be cleared when PyObject_RichCompareBool is called.
2022-09-30 12:57:09 -07:00
Guido van Rossum e9d63760fe
GH-96827: Don't touch closed loops from executor threads (#96837)
* When chaining futures, skip callback if loop closed.
* When shutting down an executor, don't wake a closed loop.
2022-09-30 12:55:40 -07:00
Victor Stinner 9f2f1dd131
gh-94526: getpath_dirname() no longer encodes the path (#97645)
Fix the Python path configuration used to initialized sys.path at
Python startup. Paths are no longer encoded to UTF-8/strict to avoid
encoding errors if it contains surrogate characters (bytes paths are
decoded with the surrogateescape error handler).

getpath_basename() and getpath_dirname() functions no longer encode
the path to UTF-8/strict, but work directly on Unicode strings. These
functions now use PyUnicode_FindChar() and PyUnicode_Substring() on
the Unicode path, rather than strrchr() on the encoded bytes string.
2022-09-30 14:58:30 +02:00
Steve Dower ff54dd96cb
gh-90989: Install Windows launcher per-user, and clarify some installer text (GH-97655) 2022-09-30 10:29:31 +01:00
Steve Dower 73942e475c
gh-97649: The Tools directory is no longer installed on Windows (GH-97653) 2022-09-30 10:25:00 +01:00
Eddie Hebert 86a3be207d
Use SyntaxError invalid range in tutorial introduction example (GH-93031)
Use output from a 3.10+ REPL, showing invalid range, for the
SyntaxError examples in the tutorial introduction page.

Automerge-Triggered-By: GH:iritkatriel
2022-09-30 01:59:46 -07:00
Ofey Chan 83a3de4e06
gh-96348: Deprecate the 3-arg signature of coroutine.throw and generator.throw (GH-96428) 2022-09-30 09:43:02 +01:00
Shin-myoung-serp 4652093e1b
gh-91212: Fixed flickering when the tracer is turned off (#95129)
Fixed flickering when the tracer is turned off.
2022-09-28 23:40:51 -07:00
Victor Stinner 83a0f44ffd
gh-97612: Fix shell injection in get-remote-certificate.py (#97613)
Fix a shell code injection vulnerability in the
get-remote-certificate.py example script. The script no longer uses a
shell to run "openssl" commands. Issue reported and initial fix by
Caleb Shortt.

Remove the Windows code path to send "quit" on stdin to the "openssl
s_client" command: use DEVNULL on all platforms instead.

Co-authored-by: Caleb Shortt <caleb@rgauge.com>
2022-09-29 01:17:27 +02:00
Victor Stinner a5f092f3c4
gh-97616: list_resize() checks for integer overflow (#97617)
Fix multiplying a list by an integer (list *= int): detect the
integer overflow when the new allocated length is close to the
maximum size.  Issue reported by Jordan Limor.

list_resize() now checks for integer overflow before multiplying the
new allocated length by the list item size (sizeof(PyObject*)).
2022-09-29 00:07:07 +02:00
Kumar Aditya 575a253b5c
GH-82448: Add thread timeout for loop.shutdown_default_executor (#97561)
Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2022-09-28 10:39:42 -07:00
LiarPrincess 0c1d7a06ed
bpo-47243: Duplicate entry in 'Objects/unicodetype_db.h' (GH-32376)
Fix for duplicate 1st entry in 'Objects/unicodetype_db.h':

```c
/* a list of unique character type descriptors */
const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {
    {0, 0, 0, 0, 0, 0},
    {0, 0, 0, 0, 0, 0}, <--- HERE
    {0, 0, 0, 0, 0, 32},
    {0, 0, 0, 0, 0, 48},
    …
```

https://bugs.python.org/issue47243

Automerge-Triggered-By: GH:isidentical
2022-09-28 06:57:14 -07:00
Pablo Galindo Salgado aab01e3524
gh-96670: Raise SyntaxError when parsing NULL bytes (#97594) 2022-09-27 23:23:42 +01:00
Serhiy Storchaka adbed2d542
gh-73588: Fix generation of the default name of tkinter.Checkbutton. (GH-97547)
Previously, checkbuttons in different parent widgets could have the same
short name and share the same state if arguments "name" and "variable" are
not specified. Now they are globally unique.
2022-09-27 14:05:05 +03:00
Cyker Way 68c46ae68b
gh-97545: Make Semaphore run faster. (#97549) 2022-09-26 16:38:00 -07:00
Victor Stinner 41351662bc
gh-96848: Fix -X int_max_str_digits option parsing (#96988)
Fix command line parsing: reject "-X int_max_str_digits" option with
no value (invalid) when the PYTHONINTMAXSTRDIGITS environment
variable is set to a valid limit.
2022-09-26 17:20:08 +02:00
Kumar Aditya c8c0afc713
GH-78724: Initialize struct.Struct in __new__ (GH-94532)
Closes https://github.com/python/cpython/issues/75960
Closes https://github.com/python/cpython/issues/78724
2022-09-25 14:32:48 +01:00
Mark Dickinson f5f047aa62
gh-96735: Fix undefined behaviour in struct unpacking functions (#96739)
This PR fixes undefined behaviour in the struct module unpacking support functions `bu_longlong`, `lu_longlong`, `bu_int` and `lu_int`; thanks to @kumaraditya303 for finding these.

The fix is to accumulate the bytes in an unsigned integer type instead of a signed integer type, then to convert to the appropriate signed type. In cases where the width matches, that conversion will typically be compiled away to a no-op.
(Evidence from Godbolt: https://godbolt.org/z/5zvxodj64 .)

To make the conversions efficient, I've specialised the relevant functions for their output size: for `bu_longlong` and `lu_longlong`, this only entails checking that the output size is indeed `8`. But `bu_int` and `lu_int` were used for format sizes `2` and `4` - I've split those into two separate functions each.

No tests, because all of the affected cases are already exercised by the test suite.
2022-09-25 10:55:33 +01:00
Oscar Benjamin 817fa28f81
gh-90716: Refactor PyLong_FromString to separate concerns (GH-96808)
This is a preliminary PR to refactor `PyLong_FromString` which is currently quite messy and has spaghetti like code that mixes up different concerns as well as duplicating logic.

In particular:

- `PyLong_FromString` now only handles sign, base and prefix detection and calls a new function `long_from_string_base` to parse the main body of the string.
- The `long_from_string_base` function handles all string validation and then calls `long_from_binary_base` or a new function `long_from_non_binary_base` to construct the actual `PyLong`.
- The existing `long_from_binary_base` function is simplified by factoring duplicated logic to `long_from_string_base`.
- The new function `long_from_non_binary_base` factors out much of the code from `PyLong_FromString` including in particular the quadratic algorithm reffered to in gh-95778 so that this can be seen separately from unrelated concerns such as string validation.
2022-09-25 10:09:50 +01:00
Serhiy Storchaka ea4be278fa
gh-97519: Synchronize links in other copies of HISTORY OF THE SOFTWARE (GH-97520)
HTTP links in the "HISTORY OF THE SOFTWARE" section of Doc/license.rst
were converted to HTTPS in f62ff97f31.
But there were other copies of these links, which were left HTTP links.
2022-09-25 11:25:24 +03:00
Steve Dower f1cca801f5
gh-96965: Update Windows release to libffi 3.4.3 (GH-97512) 2022-09-23 16:07:53 +01:00
Dong-hee Na 83d84e67cd
gh-96761: Fix build process of the clang compiler for _bootstrap_python (gh-96945)
Co-authored-by: Matthias Goergens <matthias.goergens@gmail.com>
2022-09-23 11:00:44 +09:00
Kumar Aditya 282edd7b2a
GH-85760: Fix race in calling process_exited callback too early (#97009) 2022-09-22 09:43:47 -07:00
Cyker Way 24e0379624
gh-90155: Fix bug in asyncio.Semaphore and strengthen FIFO guarantee (#93222)
The main problem was that an unluckily timed task cancellation could cause
the semaphore to be stuck. There were also doubts about strict FIFO ordering
of tasks allowed to pass.

The Semaphore implementation was rewritten to be more similar to Lock.
Many tests for edge cases (including cancellation) were added.
2022-09-22 09:34:45 -07:00
Brandt Bucher 8fd2c3b75b
GH-96975: Skip incomplete frames in PyEval_GetFrame (GH-97003) 2022-09-22 09:16:52 -07:00
Dong-hee Na 10e3d398c3
gh-97005: Update libexpat from 2.4.7 to 2.4.9 (gh-97006)
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
2022-09-22 21:25:05 +09:00
Brandt Bucher dfc73b5724
GH-95921: Fix positions for some chained comparisons (GH-96968) 2022-09-20 12:22:24 -07:00
Gregory P. Smith 34de67c094
gh-96512: Update int_max_str docs to say 3.11 (#96942)
It was unknown if it'd be before 3.11.0 when creating the original
changes.  It's in 3.11rc2, so 3.11 it is.
2022-09-19 16:43:11 -07:00
Gregory P. Smith e61ca22431
gh-95865: Further reduce quote_from_bytes memory consumption (#96860)
on large input values.  Based on Dennis Sweeney's chunking idea.
2022-09-19 16:06:25 -07:00
Victor Stinner 04f4977f50
gh-96387: take_gil() resets drop request before exit (#96869)
At Python exit, sometimes a thread holding the GIL can wait forever
for a thread (usually a daemon thread) which requested to drop the
GIL, whereas the thread already exited. To fix the race condition,
the thread which requested the GIL drop now resets its request before
exiting.

take_gil() now calls RESET_GIL_DROP_REQUEST() before
PyThread_exit_thread() if it called SET_GIL_DROP_REQUEST to fix a
race condition with drop_gil().

Issue discovered and analyzed by Mingliang ZHAO.
2022-09-20 00:13:56 +02:00
Brandt Bucher c10e33ac11
GH-96864: Check for error between line and opcode events (GH-96880) 2022-09-19 14:02:24 -07:00
Matthias Görgens cbdeda8ce7
gh-96821: Fix undefined behaviour in `_testcapimodule.c` (GH-96915)
* gh-96821: Assert for demonstrating undefined behaviour

* Fix UB

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-09-19 15:59:13 +08:00
Guido van Rossum 487135a396
Revert "gh-87079: Warn on unintended signal wakeup fd override in `asyncio` (#96807)" (#96898)
This reverts commit 0587810698.
Reason: This broke buildbots (some warnings added by that commit are turned to errors in the SSL buildbot).
Repro:  ./python Lib/test/ssltests.py
2022-09-17 14:12:45 -07:00
Lysandros Nikolaou 7e36abbb78
gh-91210: Improve error message when non-default param follows default (GH-95933)
- Improve error message when parameter without a default follows one with a default
- Show same error message when positional-only params precede the default/non-default sequence
2022-09-17 10:09:28 -07:00
Michel Hidalgo 0587810698
gh-87079: Warn on unintended signal wakeup fd override in `asyncio` (#96807)
Warn on loop initialization, when setting the wakeup fd disturbs a previously set wakeup fd, and on loop closing, when upon resetting the wakeup fd, we find it has been changed by someone else.
2022-09-17 08:07:54 -07:00
Christian Heimes 0b62964b04
gh-96883: browser: include concurrent.futures (GH-96886) 2022-09-17 13:23:39 +02:00
Victor Stinner e841ffc915
gh-95778: Mention sys.set_int_max_str_digits() in error message (#96874)
When ValueError is raised if an integer is larger than the limit,
mention sys.set_int_max_str_digits() in the error message.
2022-09-16 20:04:37 +02:00
Serhiy Storchaka 426d72e7dd
gh-96052: codeop: fix handling compiler warnings in incomplete input (GH-96132)
Previously codeop.compile_command() emitted compiler warnings (SyntaxWarning or
DeprecationWarning) and raised a SyntaxError for incomplete input containing
a potentially incorrect code. Now it always returns None for incomplete input
without emitting any warnings.
2022-09-16 17:37:30 +03:00
Dong-hee Na e47b96c44f
gh-89536: Use ThinLTO policy if possible (gh-96766) 2022-09-16 19:40:05 +09:00
adphrost a41ed975e8
GH-91049: Introduce set vectorcall field API for PyFunctionObject (GH-92257)
Co-authored-by: Andrew Frost <adfrost@fb.com>
Co-authored-by: Itamar Ostricher <itamarost@gmail.com>
2022-09-15 16:42:37 +01:00
Nikita Sobolev e37ac5fbb6
gh-96751: Remove dead code from `CALL_FUNCTION_EX` opcode (GH-96752) 2022-09-15 10:33:13 +01:00
Ruan Comelli 8e9a37dde4
Fix type annotation of `pstats.FunctionProfile.ncalls` (#96741)
* fix: annotate `pstats.FunctionProfile.ncalls` as `str`

This change aligns the type annotation of `pstats.FunctionProfile.ncalls` with its runtime type.
2022-09-14 18:33:43 -07:00
Brandt Bucher a83fdf2563
GH-90997: Improve inline cache performance for MSVC (GH-96781) 2022-09-14 17:05:04 -07:00
Yusuke Kadowaki 92338f8f19
gh-77171 Support WAVE_FORMAT_EXTENSIBLE in the wave module (GH-96777)
The test file, a modified version of Lib/test/audiodata/pluck-pcm24.wav, was provided by Andrea Celletti on the bug tracker.

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
2022-09-14 14:34:40 +01:00
Benjamin Peterson fd1e477f53
closes gh-96734: Update to Unicode 15.0.0. (GH-96809) 2022-09-13 15:45:12 -07:00
Matthias Görgens 81e36f350b
gh-96678: Fix UB of null pointer arithmetic (GH-96782)
Automerge-Triggered-By: GH:pablogsal
2022-09-13 06:14:35 -07:00
Pablo Galindo Salgado 830a20eb32
gh-93503: Document the new tracing/profiling APIs in the What's new document (#96681)
Co-authored-by: Victor Stinner <vstinner@python.org>
2022-09-13 13:12:15 +01:00
Mark Shannon 12c5f328d2
GH-96754: Check whether the interpreter frame is complete before creating frame object. (GH-96776) 2022-09-13 09:25:16 +01:00
adang1345 662782e95f
gh-96729: Ensure installers built with Tools/msi/buildrelease.bat have matching UUIDs with official releases (GH-96755) 2022-09-12 15:23:13 +01:00
Christian Heimes a36235d5c7
gh-84461: Omit resource mod and getresuid funcs on Emscripten (GH-96303) 2022-09-11 10:10:00 +02:00
Dong-hee Na 8d75a13fde
gh-90751: memoryview now supports half-float (#96738)
Co-authored-by: Antoine Pitrou <antoine@python.org>
2022-09-10 22:44:10 +02:00
Mark Shannon 50a70a083d
GH-96678: Fix undefined behavior in ceval.c (#96708) 2022-09-10 01:12:06 +01:00
Brandt Bucher b4954b1a9e
GH-90230: Fix warnings and failures with --enable-pystats (GH-96622) 2022-09-09 14:42:29 -07:00
Kumar Aditya 88a7f661ca
Fix possible NULL pointer dereference in _PyThread_CurrentFrames (GH-96584) 2022-09-09 12:33:40 +01:00
Nikita Sobolev 30cc1901ef
gh-96364: Fix text signatures of `__getitem__` for `list` and `dict` (GH-96365) 2022-09-09 17:37:02 +09:00
Dennis Sweeney 569ca27293
gh-96624: Fix test_dotted_but_module_not_loaded in testpatch.py (GH-96691)
* Update test_dotted_but_module_not_loaded to reflect the move of unittest.test to test.test_unittest.
2022-09-08 23:46:13 -04:00
Guido van Rossum df50938f58
GH-46412: More efficient bool() for ndbm/_gdbmmodule (#96692) 2022-09-08 19:32:40 -07:00
finefoot c06c001b30
gh-92734: Add indentation feature to reprlib.Repr (GH-92735) 2022-09-08 13:51:44 -05:00
Mark Shannon aa3b4cf779
GH-96636: Remove all uses of NOTRACE_DISPATCH (GH-96643)
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2022-09-08 17:16:48 +01:00
philg314 b9634ac776
gh-96352: Set AttributeError context in _PyObject_GenericGetAttrWithDict (#96353) 2022-09-08 12:12:14 +01:00
Victor Stinner c580a81af9
gh-96652: Fix faulthandler chained signal without sigaction() (#96666)
Fix the faulthandler implementation of faulthandler.register(signal,
chain=True) if the sigaction() function is not available: don't call
the previous signal handler if it's NULL.
2022-09-08 12:20:22 +02:00
Michael Droettboom 8bc356a7dd
gh-96268: Fix loading invalid UTF-8 (#96270)
This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

It also fixes an off-by-one error introduced in 3.10 for the line number when the tokenizer reports bad UTF8.
2022-09-07 14:23:54 -07:00
Charlie Zhao 3e26de3c1f
gh-94781: Fix Windows projects not cleaning intermediate and output files for frozen modules (GH-96423) 2022-09-07 21:26:53 +01:00
Steve Dower de33df27aa
gh-89545: Updates platform module to use new internal _wmi module on Windows to directly query OS properties (GH-96289) 2022-09-07 21:09:20 +01:00
Steve Dower 4114bcc9ef
gh-96577: Fixes buffer overrun in _msi module (GH-96633) 2022-09-07 18:01:35 +01:00
Raymond Hettinger 3eaf70d836
GH-96465: Cache hashes for Fraction instances (GH-96483) 2022-09-07 10:31:50 -05:00
Nikita Sobolev 2fd7246e97
gh-96641: Do not expose `KeyWrapper` in `_functoolsmodule.c` (gh-96642) 2022-09-07 20:13:07 +09:00
Michael Droettboom 05692c67c5
gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (#96623) 2022-09-07 00:12:16 +01:00
Kumar Aditya 67444902a0
GH-96572: fix use after free in trace refs build mode (#96618) 2022-09-06 11:02:41 -07:00
Mark Shannon 95e271b226
GH-96612: Skip incomplete frames in tracemalloc traces. (GH-96613) 2022-09-06 17:37:47 +01:00
Mark Shannon 222f10ca2d
GH-96569: Add two NULL checks to avoid undefined behavior. (GH-96585) 2022-09-06 16:45:43 +01:00
Brandt Bucher cd0ff9bd14
GH-93911: Fix `LOAD_ATTR_PROPERTY` caches (GH-96519) 2022-09-06 12:11:38 +01:00
Steve Dower 80a9bd2e94
gh-96559: Fixes Windows launcher handling of defaults using old-style tags, and adds What's New section (GH-96595) 2022-09-05 20:06:30 +01:00
Shantanu 200c9a8da0
gh-92986: Fix ast.unparse when ImportFrom.level is None (#92992)
This doesn't happen naturally, but is allowed by the ASDL and compiler.
We don't want to change ASDL for backward compatibility reasons
(#57645, #92987)
2022-09-05 20:14:50 +03:00
Nikita Sobolev 2c7d2e8d46
gh-96587: Raise `SyntaxError` for PEP654 on older `feature_version` (#96588) 2022-09-05 17:54:09 +01:00
Dennis Sweeney 9e35d05422
gh-96538: Move some type-checking out of bisect.bisect() loops (GH-96539) 2022-09-05 01:02:29 -04:00
Pamela Fox a0ad63e70e
gh-93973: Add all_errors to asyncio.create_connection (#93974)
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
2022-09-04 18:33:50 -07:00
Mark Dickinson b126196838
gh-95778: Correctly pre-check for int-to-str conversion (#96537)
Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)

The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.

The justification for the current check. The C code check is:
```c
max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10
```

In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is:
$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$

From this it follows that
$$\frac{M}{3L} < \frac{s-1}{10}$$
hence that
$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$
So
$$2^{L(s-1)} > 10^M.$$
But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check.

<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->

Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
2022-09-04 09:21:18 -07:00
Ikko Ashimine 6adb89f50a
no-issue: Fix typo in 3.11.0a7.rst (gh-96547)
accross -> across
2022-09-04 22:17:26 +09:00
Sergey B Kirpichev 8464b754c4
gh-68163: Correct conversion of Rational instances to float (GH-25619)
* gh-68163: Correct conversion of Rational instances to float

Also document that numerator/denominator properties are instances of Integral.

Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
2022-09-04 13:15:59 +01:00
Charlie Zhao 9b9394df5f
gh-93884: Improve test coverage of `PyNumber_ToBase` (GH-93932)
Link to #93884 
* Test with some large negative and positive values(out of range of a longlong,i.e.[-2\*\*63, 2\*\*63-1])
* Test with objects of non-int type

Automerge-Triggered-By: GH:mdickinson
2022-09-04 03:37:04 -07:00
Gregory P. Smith 511ca94520
gh-95778: CVE-2020-10735: Prevent DoS by very large int() (#96499)
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.

This PR comes fresh from a pile of work done in our private PSRT security response team repo.

Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>
Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).

<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->

I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#). Much of that text wound up in the Issue. Backports PRs already exist. See the issue for links.
2022-09-02 09:35:08 -07:00
Raymond Hettinger 656167db81
Allow whitespace around a slash in fraction string inputs (GH-96496) 2022-09-02 11:10:58 -05:00
Irit Katriel 4c72517cad
gh-93554: Conditional jump opcodes only jump forward (GH-96318) 2022-09-01 21:36:47 +01:00
Anh71me 0cd33e11fe
GH-96079 Fix missing field name for _AnnotatedAlias (#96080) 2022-08-31 16:02:24 -07:00
Vinay Sajip 29f1b0bb1f
gh-89258: Add a getChildren() method to logging.Logger. (GH-96444)
Co-authored-by: Éric <merwok@netwok.org>
2022-08-31 10:50:29 +01:00
Dennis Sweeney 8ba22b90ca
gh-95865: Speed up urllib.parse.quote_from_bytes() (GH-95872) 2022-08-30 21:39:51 -04:00
Alexandru Mărășteanu 0ed778835d
gh-95149: Enhance `http.HTTPStatus` with properties that indicate the HTTP status category (GH-95453) 2022-08-30 11:11:44 -07:00
Nikita Sobolev 4217393aee
gh-95987: Fix `repr` of `Any` type subclasses (#96412) 2022-08-30 10:36:16 -07:00
Pablo Galindo Salgado 6d791a9736
gh-96143: Allow Linux perf profiler to see Python calls (GH-96123)
⚠️  ⚠️ Note for reviewers, hackers and fellow systems/low-level/compiler engineers ⚠️ ⚠️ 

If you have a lot of experience with this kind of shenanigans and want to improve the **first** version, **please make a PR against my branch** or **reach out by email** or **suggest code changes directly on GitHub**. 

If you have any **refinements or optimizations** please, wait until the first version is merged before starting hacking or proposing those so we can keep this PR productive.
2022-08-30 10:11:18 -07:00
Daniel Giger 22ed5233b7
gh-96349: fix minor performance regression initializing threading.Event (gh-96350) 2022-08-30 21:10:02 +09:00
Nikita Sobolev 75177358a6
gh-96385: Correctly raise error on `[*T, *V]` substitution (GH-96386) 2022-08-30 10:34:55 +03:00
Christian Heimes 2eb9008d72
gh-95853: Improve WASM build script (GH-96389)
- pre-build Emscripten ports and system libraries
- check for broken EMSDK versions
- use EMSDK's node for wasm32-emscripten
- warn when PKG_CONFIG_PATH is set
- add support level information
2022-08-30 07:56:26 +02:00
Christian Heimes d0b3d235db
gh-96320: WASI socket fixes (#96388)
* gh-96320: WASI socket fixes

- ignore missing functions in ``socket.__repr__``
- bundle network files with assets

* blurb
2022-08-30 06:36:11 +02:00
Kumar Aditya e5b2453e61
GH-74116: Allow multiple drain waiters for asyncio.StreamWriter (GH-94705) 2022-08-29 11:31:11 -07:00
Christian Heimes 873554ef84
gh-94682: Build and test with OpenSSL 1.1.1q (gh-94683) 2022-08-29 18:19:15 +02:00
Petr Viktorin b2714f05c5
gh-90814: Correct NEWS wording re. optional C11 features (GH-96309)
The previous wording of this entry suggests that CPython
won't work if optional compiler features are enabled.
That's not the case. The change is that we require C11 rather
than C89.

Note that PEP 7 does say "Python 3.11 and newer versions use C11
without optional features." It is correct there: that's
not a guide for users who compile Python, but for CPython devs
who must avoid the features.
2022-08-29 13:10:52 +02:00
TW 023c51d9d8
gh-69142: add %:z strftime format code (gh-95983)
datetime.isoformat generates the tzoffset with colons, but there
was no format code to make strftime output the same format.

for simplicity and consistency the %:z formatting behaves mostly
as %z, with the exception of adding colons. this includes the
dynamic behaviour of adding seconds and microseconds only when
needed (when not 0).

this fixes the still open "generate" part of this issue:

https://github.com/python/cpython/issues/69142

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-08-28 14:27:42 -07:00
Kirill e860e521ec
gh-90467: StreamReaderProtocol - add strong reference to created task (#96323) 2022-08-27 12:32:01 -07:00
Pablo Galindo Salgado 5b070c0d40
gh-95973: Add a new --with-dsymutil option to link debug information in macOS (GH-95974)
Automerge-Triggered-By: GH:pablogsal
2022-08-26 17:49:41 -07:00
Carl Friedrich Bolz-Tereick 9c197bc8bf
GH-96172 fix unicodedata.east_asian_width being wrong on unassigned code points (#96207) 2022-08-26 19:29:39 +03:00
Christian Heimes 8c1dbad36f
gh-96269: Fix build dependency on AIX (GH-96304) 2022-08-26 12:31:58 +02:00
Harry ad7340e8c5
gh-92445 Improve interaction between nargs="*" and choices() (GH-92565) 2022-08-25 06:18:38 -05:00
Sam Ezeh cd492d43a2
gh-76728: Coerce DictReader and DictWriter fieldnames argument to a list (GH-32225) 2022-08-25 05:13:24 -05:00
Mark Shannon c09fa7542c
GH-96237: Allow non-functions as reference-holder in frames. (GH-96238) 2022-08-25 10:16:55 +01:00
Ross Burton df11012697
gh-95243: Mitigate the race condition in testSockName (#96173)
find_unused_port() has an inherent race condition, but we can't use
bind_port() as that uses .getsockname() which this test is exercising.

Try binding to unused ports a few times before failing.

Signed-off-by: Ross Burton <ross.burton@arm.com>
2022-08-24 18:37:18 -07:00
Pablo Galindo Salgado e34c82abeb
GH-93503: Add thread-specific APIs to set profiling and tracing functions in the C-API (#93504)
* gh-93503: Add APIs to set profiling and tracing functions in all threads in the C-API

* Use a separate API

* Fix NEWS entry

* Add locks around the loop

* Document ignoring exceptions

* Use the new APIs in the sys module

* Update docs
2022-08-24 23:21:39 +01:00
prego 4317b25a23
GH-96179: Fix misleading example on the bisect documentation (GH-96228)
The `movies[bisect(movies, 1960, key=by_year)]` will actually return only movies **after** 1960.
2022-08-24 17:47:13 +02:00
Irit Katriel 420f39f457
gh-93678: add _testinternalcapi.optimize_cfg() and test utils for compiler optimization unit tests (GH-96007) 2022-08-24 11:02:53 +01:00
Raymond Hettinger 1f0eafa844
GH-96145: Add AttrDict to JSON module for use with object_hook (#96146) 2022-08-23 16:22:00 -05:00
Kevin Kirsche 58f6953d6d
gh-96175: add missing self._localName assignment in `xml.dom.minidom.Attr` (#96176)
X-Ref: https://github.com/python/typeshed/pull/8590#discussion_r951473977

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-08-23 09:16:02 -07:00
Pablo Galindo Salgado 16ebae4cd4
GH-96187: Prevent _PyCode_GetExtra to return garbage for negative indexes (GH-96188) 2022-08-23 11:13:53 +01:00
Duncan Grisby 1499d73b3e
gh-96159: Fix significant performance degradation in logging.TimedRotat… (GH-96182) 2022-08-23 07:28:43 +01:00
Christian Heimes 53e6a9a725
gh-96046: Initialize ht_cached_keys in PyType_Ready() (GH-96047) 2022-08-21 22:24:03 -07:00
Nick Coghlan bcc4cb0c7d
gh-96098: Clearly link concurrent.futures from threading & multiprocessing docs (GH-96112)
Clearly link concurrent.futures from threading & multiprocessing docs

Also link directly to asyncio from the beginning of the threading docs.
2022-08-21 14:09:05 +12:00
Kumar Aditya e0d54a4a79
GH-96071: fix deadlock in PyGILState_Ensure (GH-96124)
Alternative of #96107
2022-08-19 12:43:00 -07:00
Christian Heimes 822955c166
gh-96125: Fix sys.thread_info.name on pthread platforms (GH-96126)
Automerge-Triggered-By: GH:tiran
2022-08-19 12:41:25 -07:00
Brandt Bucher 5bfb3c372b
GH-90997: Wrap yield from/await in a virtual try/except StopIteration (GH-96010) 2022-08-19 12:33:44 -07:00
Carl Friedrich Bolz-Tereick 2d9f252c0c
gh-96019: Fix caching of decompositions in makeunicodedata (GH-96020) 2022-08-19 12:20:44 +03:00
Pablo Galindo Salgado 9d066e2aa6
gh-95463: Remove backwards incompatible change regarding the _MASK_UTF_FILENAME flags in bpo-28080 (GH-96072)
Automerge-Triggered-By: GH:pablogsal
2022-08-18 16:45:55 -07:00
Kevin Modzelewski 214eb2cce5
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer

Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.

It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).

Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.

The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).

The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.

This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.

* Simplify the build flags

* Add a NEWS entry

* Update Makefile.pre.in

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Update configure.ac

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Add myself to ACKS

* Add docs

* Other review comments

* fix tab/space issue

* Make it more clear that --enable-bolt is experimental

* Add link to bolt's github page

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-19 06:33:54 +09:00
Raymond Hettinger 29c8f80760
GH-95861: Add support for Spearman's rank correlation coefficient (GH-95863) 2022-08-18 13:48:27 -05:00
Mark Shannon d8c07f8cb4
Correct news entry. (GH-96043) 2022-08-17 12:59:44 +01:00
Mark Shannon b73e3b6d4a
GH-95589: Dont crash when subclassing extension classes with multiple inheritance (GH-96028)
* Treat tp_weakref and tp_dictoffset like other opaque slots for multiple inheritance.

* Document Py_TPFLAGS_MANAGED_DICT and Py_TPFLAGS_MANAGED_WEAKREF in what's new.
2022-08-17 12:50:53 +01:00
Ken Jin 7276ca25f5
GH-93911: Specialize `LOAD_ATTR` for custom `__getattribute__` (GH-93988) 2022-08-17 12:37:07 +01:00
Guido van Rossum f51f54f39d
GH-95704: Don't suppress errors from tasks when TG is cancelled (#95761)
When a task catches CancelledError and raises some other error,
the other error should not silently be suppressed.

Any scenario where a task crashes in cleanup upon cancellation
will now result in an ExceptionGroup wrapping the crash(es)
instead of propagating CancelledError and ignoring the side errors.

NOTE: This represents a change in behavior (hence the need to
change several tests).  But it is only an edge case.

Co-authored-by: Thomas Grainger <tagrain@gmail.com>
2022-08-16 18:23:06 -07:00
Christian Heimes 48174fa0b9
gh-96005: Handle WASI ENOTCAPABLE in getpath (GH-96006)
- On WASI `ENOTCAPABLE` is now mapped to `PermissionError`.
- The `errno` modules exposes the new error number.
- `getpath.py` now ignores `PermissionError` when it cannot open landmark
  files `pybuilddir.txt` and `pyenv.cfg`.
2022-08-16 20:20:15 +02:00
Kumar Aditya 9d515997f9
GH-95736: fix IsolatedAsyncioTestCase to initialize Runner before calling setup functions (#95898) 2022-08-16 08:52:06 -07:00
Mark Shannon 5a8c15819c
GH-95245: Move weakreflist into the pre-header. (GH-95996) 2022-08-16 13:57:18 +01:00
Christian Heimes ab4d72954f
gh-95957: Add instructions for Tcl/Tk and OpenSSL on RHEL/CentOS 7 (#95964)
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-08-16 13:44:02 +02:00
Kumar Aditya 914f6367a0
GH-95899: fix asyncio.Runner to call set_event_loop only once (#95900) 2022-08-15 10:02:47 -07:00
Shreenidhi Shedi 2fa03b1b07
gh-95231: Disable md5 & crypt modules if FIPS is enabled (GH-94742)
If kernel fips is enabled, we get permission error upon doing
`import crypt`. So, if kernel fips is enabled, disable the
unallowed hashing methods.

Python 3.9.1 (default, May 10 2022, 11:36:26)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import crypt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/crypt.py", line 117, in <module>
    _add_method('MD5', '1', 8, 34)
  File "/usr/lib/python3.9/crypt.py", line 94, in _add_method
    result = crypt('', salt)
  File "/usr/lib/python3.9/crypt.py", line 82, in crypt
    return _crypt.crypt(word, salt)
PermissionError: [Errno 1] Operation not permitted

Signed-off-by: Shreenidhi Shedi <sshedi@vmware.com>
2022-08-15 07:48:07 -07:00
Mark Shannon 3ef3c6306d
GH-95707: Fix uses of `Py_TPFLAGS_MANAGED_DICT` (GH-95854)
* Make sure that tp_dictoffset is correct with Py_TPFLAGS_MANAGED_DICT is set.

* Avoid traversing managed dict twice when subclassing class with Py_TPFLAGS_MANAGED_DICT set.
2022-08-15 12:29:27 +01:00
Kumar Aditya f2afdf3352
GH-95977: Speed up calling pure python descriptor __get__ with vectorcall (gh-95978) 2022-08-15 00:13:42 +09:00
Christian Heimes 32ac98e899
gh-95853: Add script to automate WASM build (GH-95828)
Automate WASM build with a new Python script. The script provides
several build profiles with configure flags for Emscripten flavors
and WASI. The script can detect and use Emscripten SDK and WASI SDK from
default locations or env vars.

``configure`` now detects Node arguments and creates HOSTRUNNER
arguments for Node 16. It also sets correct arguments for
``wasm64-emscripten``.

Co-authored-by: Brett Cannon <brett@python.org>
2022-08-13 21:56:08 +02:00
Barney Gale 187949ebf2
gh-94909: fix joining of absolute and relative Windows paths in pathlib (GH-95450)
Have pathlib use `os.path.join()` to join arguments to the `PurePath` initialiser, which fixes a minor bug when handling relative paths with drives.

Previously:

```python
>>> from pathlib import PureWindowsPath
>>> a = 'C:/a/b'
>>> b = 'C:x/y'
>>> PureWindowsPath(a, b)
PureWindowsPath('C:x/y')
```

Now:

```python
>>> PureWindowsPath(a, b)
PureWindowsPath('C:/a/b/x/y')
```
2022-08-12 14:23:41 -07:00
Shantanu b5e3ea2862
gh-94996: Disallow parsing pos only params with feature_version < (3, 8) (GH-94997) 2022-08-12 19:27:50 +02:00
Irit Katriel 41757bfabd
gh-95922: compiler's eliminate_empty_basic_blocks ignores the last block of the compilation unit (GH-95924) 2022-08-12 16:35:09 +01:00
Mark Shannon 1b46d118e6
GH-95818: Skip incomplete frames in `PyThreadState_GetFrame` (GH-95886) 2022-08-11 14:06:32 +01:00
Christian Heimes b4c857d0fd
gh-95876: Fix format string in pegen error location code (#95877) 2022-08-11 09:55:57 +01:00
Steve Dower 73d8ffefe9
gh-95733: Allow installing Store package on older Windows versions (GH-95862) 2022-08-11 00:47:58 +01:00
Mark Dickinson 97e9cfa75a
gh-95605: Fix `float(s)` error message when `s` contains only whitespace (GH-95665)
This PR fixes the error message from float(s) in the case where s contains only whitespace.
2022-08-10 19:25:39 +01:00
David Bonner 37c0f9ccc0
gh-95804: Respect MemoryHandler.flushOnClose in logging shutdown. (GH-95857) 2022-08-10 18:08:55 +01:00
Petr Viktorin 71c3d649b5
gh-95504: Fix negative numbers in PyUnicode_FromFormat (GH-95848)
Co-authored-by: philg314 <110174000+philg314@users.noreply.github.com>
2022-08-10 13:12:40 +02:00
Petr Viktorin 325ae93b6b
gh-93649: Split unicode tests from _testcapimodule.c & add some more (GH-95819)
- Move PyUnicode tests to a separate file
- Add some more tests for PyUnicode_FromFormat

Co-authored-by: philg314 <110174000+philg314@users.noreply.github.com>
2022-08-10 09:10:25 +02:00
Serhiy Storchaka 62f06508e7
gh-95781: More strict format string checking in PyUnicode_FromFormatV() (GH-95784)
An unrecognized format character in PyUnicode_FromFormat() and
PyUnicode_FromFormatV() now sets a SystemError.
In previous versions it caused all the rest of the format string to be
copied as-is to the result string, and any extra arguments discarded.
2022-08-08 19:21:07 +03:00
Petr Viktorin 656dad702d
gh-93274: Expose receiving vectorcall in the Limited API (GH-95717) 2022-08-08 14:12:05 +02:00