* as_completed returns object that is both iterator and async iterator
* Existing tests adjusted to test both the old and new style
* New test to ensure iterator can be resumed
* New test to ensure async iterator yields any passed-in Futures as-is
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
* Reads zip64 files as produced by the zipfile module
* Include tests (somewhat slow, however, because of the need to create "large" zips)
* About the same amount of strictness reading invalid zip files as zipfile has
* Still works on files with prepended data (like pex)
There are a lot more test cases at https://github.com/thatch/zipimport64/ that give me confidence that this works for real-world files.
Fixes#89739 and #77140.
---------
Co-authored-by: Itamar Ostricher <itamarost@gmail.com>
Reviewed-by: Gregory P. Smith <greg@krypto.org>
* GH-113171: Fix "private" (really non-global) IP address ranges
The _private_networks variables, used by various is_private
implementations, were missing some ranges and at the same time had
overly strict ranges (where there are more specific ranges considered
globally reachable by the IANA registries).
This patch updates the ranges with what was missing or otherwise
incorrect.
I left 100.64.0.0/10 alone, for now, as it's been made special in [1]
and I'm not sure if we want to undo that as I don't quite understand the
motivation behind it.
The _address_exclude_many() call returns 8 networks for IPv4, 121
networks for IPv6.
[1] https://github.com/python/cpython/issues/61602
Add Py_GetConstant() and Py_GetConstantBorrowed() functions.
In the limited C API version 3.13, getting Py_None, Py_False,
Py_True, Py_Ellipsis and Py_NotImplemented singletons is now
implemented as function calls at the stable ABI level to hide
implementation details. Getting these constants still return borrowed
references.
Add _testlimitedcapi/object.c and test_capi/test_object.py to test
Py_GetConstant() and Py_GetConstantBorrowed() functions.
These give applications the option of more forcefully terminating client
connections for asyncio servers. Useful when terminating a service and
there is limited time to wait for clients to finish up their work.
This is a do-over with a test fix for gh-114432, which was reverted.
On Windows, time.monotonic() now uses the QueryPerformanceCounter()
clock to have a resolution better than 1 us, instead of the
gGetTickCount64() clock which has a resolution of 15.6 ms.
These give applications the option of more forcefully terminating client
connections for asyncio servers. Useful when terminating a service and
there is limited time to wait for clients to finish up their work.
This adds `VERIFY_X509_STRICT` to make the default
SSL context perform stricter (per RFC 5280) validation, as well
as `VERIFY_X509_PARTIAL_CHAIN` to enforce more standards-compliant
path-building behavior.
As part of this changeset, I had to tweak `make_ssl_certs.py`
slightly to emit 5280-conforming CA certs. This changeset includes
the regenerated certificates after that change.
Signed-off-by: William Woodruff <william@yossarian.net>
Co-authored-by: Victor Stinner <vstinner@python.org>
Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:
- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`
Based on the "flush" idea from https://github.com/python/cpython/pull/115138#issuecomment-1932444270 .
### Notes
- Please treat as a security fix related to CVE-2023-52425.
Includes code suggested-by: Snild Dolkow <snild@sony.com>
and by core dev Serhiy Storchaka.
Add PythonFinalizationError exception. This exception derived from
RuntimeError is raised when an operation is blocked during the Python
finalization.
The following functions now raise PythonFinalizationError, instead of
RuntimeError:
* _thread.start_new_thread()
* subprocess.Popen
* os.fork()
* os.fork1()
* os.forkpty()
Morever, _winapi.Overlapped finalizer now logs an unraisable
PythonFinalizationError, instead of an unraisable RuntimeError.
By default, it preserves an inconsistent behavior of older Python
versions: packs the count into a 1-tuple if only one or none
options are specified (including 'update'), returns None instead of 0.
Except that setting wantobjects to 0 no longer affects the result.
Add a new parameter return_ints: specifying return_ints=True makes
Text.count() always returning the single count as an integer
instead of a 1-tuple or None.
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add optional 'filter' parameter to iterdump() that allows a "LIKE"
pattern for filtering database objects to dump.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
* When called with a single argument to get a value, it allow to omit
the minus prefix.
* It can be called with keyword arguments to set attributes.
* w.wm_attributes(return_python_dict=True) returns a dict instead of
a tuple (it will be the default in future).
* Setting wantobjects to 0 no longer affects the result.
The new `PyList_GetItemRef` is similar to `PyList_GetItem`, but returns
a strong reference instead of a borrowed reference. Additionally, if the
passed "list" object is not a list, the function sets a `TypeError`
instead of calling `PyErr_BadInternalCall()`.
---------
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Return files and directories from `pathlib.Path.glob()` if the pattern ends
with `**`. This is more compatible with `PurePath.full_match()` and with
other glob implementations such as bash and `glob.glob()`. Users can add a
trailing slash to match only directories.
In my previous patch I added a `FutureWarning` with the intention of fixing
this in Python 3.15. Upon further reflection I think this was an
unnecessarily cautious remedy to a clear bug.
Add `ntpath.isreserved()`, which identifies reserved pathnames such as "NUL", "AUX" and "CON".
Deprecate `pathlib.PurePath.is_reserved()`.
---------
Co-authored-by: Eryk Sun <eryksun@gmail.com>
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
In 49f90ba we added support for the recursive wildcard `**` in
`pathlib.PurePath.match()`. This should allow arbitrary prefix and suffix
matching, like `p.match('foo/**')` or `p.match('**/foo')`, but there's a
problem: for relative patterns only, `match()` implicitly inserts a `**`
token on the left hand side, causing all patterns to match from the right.
As a result, it's impossible to match relative patterns from the left:
`PurePath('foo/bar').match('bar/**')` is true!
This commit reverts the changes to `match()`, and instead adds a new
`full_match()` method that:
- Allows empty patterns
- Supports the recursive wildcard `**`
- Matches the *entire* path when given a relative pattern
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
On Windows, `os.path.isabs()` now returns `False` when given a path that
starts with exactly one (back)slash. This is more compatible with other
functions in `os.path`, and with Microsoft's own documentation.
Also adjust `pathlib.PureWindowsPath.is_absolute()` to call
`ntpath.isabs()`, which corrects its handling of partial UNC/device paths
like `//foo`.
Co-authored-by: Jon Foster <jon@jon-foster.co.uk>
It can be used to set the location of a .python_history file
---------
Co-authored-by: Levi Sabah <0xl3vi@gmail.com>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Sync with importlib_metadata 7.0.0
* Add blurb
* Update docs to reflect changes.
* Link datamodel docs for object.__getitem__
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* Add what's new for removed __getattr__
* Link datamodel docs for object.__getitem__
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* Add exclamation point, as that seems to be used for other classes.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Add support for `os.POSIX_SPAWN_CLOSEFROM` and
`posix_spawn_file_actions_addclosefrom_np` and have the `subprocess` module use
them when available. This means `posix_spawn` can now be used in the default
`close_fds=True` situation on many platforms.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
* Allow posix_spawn to inherit environment form parent environ variable.
With this change, posix_spawn call can behave similarly to execv with regards to environments when used in subprocess functions.
PR #100161 added fancy float-style formatting for the Fraction type,
but left us in a state where basic formatting for fractions (alignment,
fill, minimum width, thousands separators) still wasn't supported.
This PR adds that support.
---------
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Detect email address parsing errors and return empty tuple to
indicate the parsing error (old API). Add an optional 'strict'
parameter to getaddresses() and parseaddr() functions. Patch by
Thomas Dwyer.
Co-Authored-By: Thomas Dwyer <github@tomd.tel>
Renamed re.error for clarity, and kept re.error for backward compatibility.
Updated idlelib files at TJR's request.
---------
Co-authored-by: Matthias Bussonnier <mbussonnier@ucmerced.edu>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
As of gh-112661, the threading module expects the _thread module to have a _is_main_interpreter(), which is used in the internal threading._shutdown(). This change causes a problem for anyone that replaces the _thread module with a custom one (only if they don't provide _is_main_interpreter()). They need to be sure to add it for 3.13+, thus this PR is adding a note in "What's New".
This also forward-ports the "What's New" entry from 3.12 (gh-112850). Note that we do not also forward-port the fix in that PR. The fix is there only due to a regression from 3.12.0. There is no regression in 3.13+.
* Implement _Py_HashPointerRaw() as a static inline function.
* Add Py_HashPointer() tests to test_capi.test_hash.
* Keep _Py_HashPointer() function as an alias to Py_HashPointer().
Change the declaration of the keywords parameter in functions
PyArg_ParseTupleAndKeywords() and PyArg_VaParseTupleAndKeywords() from `char **`
to `char * const *` in C and `const char * const *` in C++.
It makes these functions compatible with argument of type `const char * const *`,
`const char **` or `char * const *` in C++ and `char * const *` in C
without explicit type cast.
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
* bpo-32731: Raise OSError from any failure in getpass.getuser()
Previously, if the username was not set in certain environment variables, ImportError escaped on Windows systems, and it was possible for KeyError to escape on other systems if getpwuid() failed.
_PyDict_Pop_KnownHash(): remove the default value and the return type
becomes an int.
Co-authored-by: Stefan Behnel <stefan_ml@behnel.de>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Add `glob.translate()` function that converts a pathname with shell wildcards to a regular expression. The regular expression is used by pathlib to implement `match()` and `glob()`.
This function differs from `fnmatch.translate()` in that wildcards do not match path separators by default, and that a `*` pattern segment matches precisely one path segment. When *recursive* is set to true, `**` pattern segments match any number of path segments, and `**` cannot appear outside its own segment.
In pathlib, this change speeds up directory walking (because `_make_child_relpath()` does less work), makes path objects smaller (they don't need a `_lines` slot), and removes the need for some gnarly code.
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
* Split list_extend() into two sub-functions: list_extend_fast() and
list_extend_iter().
* list_inplace_concat() no longer has to call Py_DECREF() on the
list_extend() result, since list_extend() now returns an int.
* Revert "gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (#111585)"
This reverts commit d9b606b3d0.
* Revert "gh-111089: Use PyUnicode_AsUTF8() in getargs.c (#111620)"
This reverts commit cde1071b2a.
* Revert "gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (#111091)"
This reverts commit d731579bfb.
* Revert "gh-111089: Add PyUnicode_AsUTF8() to the limited C API (#111121)"
This reverts commit d8f32be5b6.
* Revert "gh-111089: Use PyUnicode_AsUTF8() in sqlite3 (#111122)"
This reverts commit 37e4e20eaa.
gh-106168: Update the size only after setting the item, to avoid temporary inconsistencies.
Also remove the "what's new" sentence regarding the size setting since tuples cannot grow after allocation.
It now always returns an integer if one or less counting options are specified.
Previously it could return a single count as a 1-tuple, an integer (only if
option "update" was specified) or None if no items found.
The result is now the same if wantobjects is set to 0.
Add PyUnicode_AsUTF8() function to the limited C API.
multiprocessing posixshmem now uses PyUnicode_AsUTF8() instead of
PyUnicode_AsUTF8AndSize(): the extension is built with the limited C
API. The function now raises an exception if the filename contains an
embedded null character instead of truncating silently the filename.
* PyUnicode_AsUTF8() now raises an exception if the string contains
embedded null characters.
* Update related C API tests (test_capi.test_unicode).
* type_new_set_doc() uses PyUnicode_AsUTF8AndSize() to silently
truncate doc containing null bytes.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Build the _multiprocessing.posixshmem extension with the Limited C
API.
* Add <errno.h> include.
* Replace PyUnicode_AsUTF8() with PyUnicode_AsUTF8AndSize().
* Replace PyStructSequence_SET_ITEM() with
PyStructSequence_SetItem().
* Replace PyTuple_GET_SIZE() with PyTuple_Size().
* Replace PyTuple_GET_ITEM() with PyTuple_GetItem().
The PySys_Audit() function was added in Python 3.8 by the PEP 578
"Python Runtime Audit Hooks".
Add also PySys_AuditTuple() to the limited C API, function added
to Python 3.13.
Move non-limited "PerfMap" C API from Include/sysmodule.h to
Include/cpython/sysmodule.h.
Add PyMem_RawMalloc(), PyMem_RawCalloc(), PyMem_RawRealloc() and
PyMem_RawFree() to the limited C API.
These functions were added by Python 3.4 and are needed to port
stdlib extensions to the limited C API, like grp and pwd.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
It already mostly worked, except in the case when invalid keyword
argument with non-ASCII name was passed to function with non-ASCII
parameter names. Then it crashed in the debug mode.
If the Py_LIMITED_API macro is defined, Py_BUILD_CORE,
Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE macros are now
undefined by Python.h.
Only undefine these 3 macros after including "exports.h" which uses
them to define PyAPI_FUNC(), PyAPI_DATA() and PyMODINIT_FUNC macros.
Remove hacks (undefine manually the 3 Py_BUILD_CORE macros) in
Modules/_testcapi/parts.h and Modules/_testclinic_limited.c.
Add wrapper for timerfd_create, timerfd_settime, and timerfd_gettime to os module.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
sys.audit() now has assertions to check that the event argument is
not NULL and that the format argument does not use the "N" format.
Add tests on PySys_AuditTuple().
* Use `FindFirstFile` Win32 API to fix a bug where `ntpath.realpath()`
breaks out of traversing a series of paths where a (handled)
`ERROR_ACCESS_DENIED` or `ERROR_SHARING_VIOLATION` occurs.
* Update docs to reflect that `ntpath.realpath()` eliminates MS-DOS
style names.
Add PyThreadState_GetUnchecked() function: similar to
PyThreadState_Get(), but don't issue a fatal error if it is NULL. The
caller is responsible to check if the result is NULL. Previously,
this function was private and known as _PyThreadState_UncheckedGet().
Make PyObject_VisitManagedDict() and PyObject_ClearManagedDict()
functions public in Python 3.13 C API.
* Rename _PyObject_VisitManagedDict() to PyObject_VisitManagedDict().
* Rename _PyObject_ClearManagedDict() to PyObject_ClearManagedDict().
* Document these functions.
If the timeout is greater than PY_TIMEOUT_MAX,
PyThread_acquire_lock_timed() uses a timeout of PY_TIMEOUT_MAX
microseconds, which is around 280.6 years. This case is unlikely and
limiting a timeout to 280.6 years sounds like a reasonable trade-off.
The constant PY_TIMEOUT_MAX is not used in PyPI top 5,000 projects.
This method supports file URIs (including variants) as described in RFC 8089, such as URIs generated by `pathlib.Path.as_uri()` and `urllib.request.pathname2url()`.
The method is added to `Path` rather than `PurePath` because it uses `os.fsdecode()`, and so its results vary from system to system. I intend to deprecate `PurePath.as_uri()` and move it to `Path` for the same reason.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
* Refactor os_sched_getaffinity_impl(): move variable definitions to
their first assignment.
* Fix test_posix.test_sched_getaffinity(): restore the old CPU mask
when the test completes!
* Doc: Specify that os.cpu_count() counts *logicial* CPUs.
* Doc: Specify that os.sched_getaffinity(0) is related to the calling
thread.
* Remove unused <locale.h> includes.
* Remove unused <fcntl.h> include in traceback.h.
* Remove redundant <assert.h> and <stddef.h> includes. They are already
included by "Python.h".
* Remove <object.h> include in faulthandler.c. Python.h already includes it.
* Add missing <stdbool.h> in pycore_pythread.h if HAVE_PTHREAD_STUBS
is defined.
* Fix also warnings in pthread_stubs.h: don't redefine macros if they
are already defined, like the __NEED_pthread_t macro.
* pycore_pythread.h is now the central place to make sure that
_POSIX_THREADS and _POSIX_SEMAPHORES macros are defined if
available.
* Make sure that pycore_pythread.h is included when _POSIX_THREADS
and _POSIX_SEMAPHORES macros are tested.
* PY_TIMEOUT_MAX is now defined as a constant, since its value
depends on _POSIX_THREADS, instead of being defined as a macro.
* Prevent integer overflow in the preprocessor when computing
PY_TIMEOUT_MAX_VALUE on Windows:
replace "0xFFFFFFFELL * 1000 < LLONG_MAX"
with "0xFFFFFFFELL < LLONG_MAX / 1000".
* Document the change and give hints how to fix affected code.
* Add an exception for PY_TIMEOUT_MAX name to smelly.py
* Add PY_TIMEOUT_MAX to the stable ABI
Justin Turner Arthur
Barney Gale
Pedro Lacerda
Tim Hatch
Victor Stinner
Terry Jan Reedy
Serhiy Storchaka
Jakub Stasiak
Mark Shannon
Bogdan Romanyuk
Pierre Ossman (ThinLinc team)
Nikita Sobolev
Guido van Rossum
William Woodruff
Jason Zhang
pan324
Sebastian Pipping
Erlend E. Aasland
Jelle Zijlstra
Hugo van Kemenade
Raymond Hettinger
Matan Perelman
Jamie Phan
Steve Dower
Petr Viktorin
Mike Zimin
Laurie O
Brett Cannon
Shantanu
Edgar Ramírez Mondragón
Sam Gross
Mariusz Felisiak
technillogue
Karolina Surma
Zackery Spytz
Alex Waygood
Hugo van Kemenade
Parth Doshi
Inada Naoki
Jason R. Coombs
Jakub Kulík
Mark Dickinson
Miro Hrončok
achhina
Eric Snow
Pablo Galindo Salgado
Kamil Turek
Irit Katriel
Matt Prodani
Tian Gao
Jacob Walls
Adam Turner
Lincoln
Ori Avtalion
Donghee Na
Charles Machalow
scoder
Yilei Yang
Pablo Martí Gamboa
Łukasz Langa
Ezio Melotti
Masaru Tsuchiyama
박문식
Lele Gaifax
numbermaniac