Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:
- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`
Based on the "flush" idea from https://github.com/python/cpython/pull/115138#issuecomment-1932444270 .
### Notes
- Please treat as a security fix related to CVE-2023-52425.
Includes code suggested-by: Snild Dolkow <snild@sony.com>
and by core dev Serhiy Storchaka.
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Introduce a new subsubsection, 'Functions', for module level functions,
and place it before the PrettyPrinter class reference.
Also:
- Fix pprint.pprint() references so they properly link to the module
level function.
- Add links to sys.stdout.
Remove a left-over sentence that refers to Py_OptimizeFlag
Remove a left-over sentence that refers to an example that was present in Python 3.10 and was using ``Py_OptimizeFlag``.
Explain the `full_match()` / `glob()` / `rglob()` pattern language in its own section. Move `rglob()` documentation under `glob()` and reduce duplicated text.
* clean up fcntl module doc
* simplify
* a few changes, based on suggestion by CAM-Gerlach
* nitpick ignore for a couple other C functions mentioned in the fcntl module doc
* more changes, especially related to LOCK_* constants
* :data: back to :const:
* Apply suggestions from code review
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
---------
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Content adapted from https://devguide.python.org/development-tools/gdb/#
and https://wiki.python.org/moin/DebuggingWithGdb.
The original content on the Wiki page came from gdb debug help used by
the Launchpad (https://launchpad.net/) team.
Thanks to Anatoly Techtonik and user `rmf` for substantial improvements to the Wiki page.
The history of the Devguide page follows
(with log entries expanded for major content contributions):
Hugo van Kemenade, Sat Dec 30 21:22:04 2023 +0200
Hugo van Kemenade, Fri Dec 8 12:04:32 2023 +0200
Erlend E. Aasland & Hugo van Kemenade, Tue Aug 8 22:05:34 2023 +0200
Satish Mishra, Sat Feb 11 13:54:57 2023 +0530
Hugo van Kemenade, Fri Dec 23 17:33:33 2022 +0200
Skip Montanaro, Hugo, Erlend, & Ezio, Fri Nov 4 05:04:23 2022 -0500
Add a GDB tips section to Advanced Tools (#977)
Adam Turner, Wed Jun 15 21:19:23 2022 +0100
Adam Turner, Tue Jun 14 11:12:26 2022 +0100
Suriyaa, Fri Jun 8 19:39:23 2018 +0200
Jeff Allen, Tue Oct 24 18:12:53 2017 +0100
Jeff Allen, Fri Oct 13 13:43:43 2017 +0100
Mariatta, Wed Jan 4 09:14:55 2017 -0800
Carol Willing, Mon Sep 26 14:50:54 2016 -0700
Zachary Ware, Thu Jul 21 10:42:23 2016 -0500
Georg Brandl, Mon Nov 3 11:28:19 2014 +0100
Add instruction how to activate python-gdb.py
Georg Brandl, Sun Mar 9 10:32:01 2014 +0100
Georg Brandl, Tue Apr 3 09:12:53 2012 +0200
Georg Brandl, Sat Mar 5 17:32:35 2011 +0100
Dave Malcolm, Fri Jan 21 12:34:09 2011 -0500
Add documentation on the gdb extension commands provided in libpython.py
I adapted this from documentation I wrote for the Fedora wiki:
https://fedoraproject.org/wiki/Features/EasierPythonDebugging#New_gdb_commands
reformatting it as rst, and making other minor changes
Brett Cannon, Thu Jan 20 15:16:52 2011 -0800
Dave Malcolm, Thu Jan 20 16:17:23 2011 -0500
Add some notes on the gdb pretty-printer hooks
Antoine Pitrou, Thu Jan 20 21:17:49 2011 +0100
Give an example backtrace
Antoine Pitrou, Thu Jan 20 21:03:06 2011 +0100
Expand explanations about gdb support
Brett Cannon, Thu Jan 20 11:33:36 2011 -0800
Tweak the gdb support title to fit in better with the devguide.
Brett Cannon, Mon Jan 17 21:12:54 2011 +0000
Short README on gdb support.
Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>
Co-authored-by: anatoly techtonik <techtonik@gmail.com>
Co-authored-by: Antoine Pitrou <solipsis@pitrou.net>
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Dave Malcolm <dmalcolm@redhat.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Co-authored-by: Georg Brandl <georg@python.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Jeff Allen <ja.py@farowl.co.uk>
Co-authored-by: Mariatta <Mariatta@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Satish Mishra <7506satish@gmail.com>
Co-authored-by: Skip Montanaro <skip.montanaro@gmail.com>
Co-authored-by: Suriyaa <isc.suriyaa@gmail.com>
Co-authored-by: Zachary Ware <zachary.ware@gmail.com>
This expands the examples to cover both realistic use cases for the API.
I noticed thing in the test that could be done better so I added those as well: We need to guarantee that all bytes of the result are overwritten and that too many are not written. Tests now pre-fills the result with data in order to ensure that.
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
The charset name "Windows-31J" is registered in the IANA Charset Registry[1]
and is implemented in Python as the cp932 codec.
[1] https://www.iana.org/assignments/charset-reg/windows-31J
Signed-off-by: Masayuki Moriyama <masayuki.moriyama@miraclelinux.com>
Fix the exceptions raised by posixpath.commonpath
Raise ValueError, not IndexError when passed an empty iterable. Raise
TypeError, not ValueError when passed None.
Update documentation for re library to explain that a backreference `\g<0>` is
expanded to the entire string when using Match.expand().
Note that numeric backreferences to group 0 (`\0`) are not supported.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
This adds a safe memory reclamation scheme based on FreeBSD's "GUS" and
quiescent state based reclamation (QSBR). The API provides a mechanism
for callers to detect when it is safe to free memory that may be
concurrently accessed by readers.
* bpo-38364: unwrap partialmethods just like we unwrap partials
The inspect.isgeneratorfunction, inspect.iscoroutinefunction and inspect.isasyncgenfunction already unwrap functools.partial objects, this patch adds support for partialmethod objects as well.
Also: Rename _partialmethod to __partialmethod__.
Since we're checking this attribute on arbitrary function-like objects,
we should use the namespace reserved for core Python.
---------
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add PythonFinalizationError exception. This exception derived from
RuntimeError is raised when an operation is blocked during the Python
finalization.
The following functions now raise PythonFinalizationError, instead of
RuntimeError:
* _thread.start_new_thread()
* subprocess.Popen
* os.fork()
* os.fork1()
* os.forkpty()
Morever, _winapi.Overlapped finalizer now logs an unraisable
PythonFinalizationError, instead of an unraisable RuntimeError.
Keep the page though, because people might still rely on it (the traffic shows that they do).
Instead of our own manual we now give links to the 3rd-party ones.
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Now the special comparison methods like `__eq__` and `__lt__` return
NotImplemented if one of comparands is date and other is datetime
instead of ignoring the time part and the time zone or forcefully
return "not equal" or raise TypeError.
It makes comparison of date and datetime subclasses more symmetric
and allows to change the default behavior by overriding
the special comparison methods in subclasses.
It is now the same as if date and datetime was independent classes.
By default, it preserves an inconsistent behavior of older Python
versions: packs the count into a 1-tuple if only one or none
options are specified (including 'update'), returns None instead of 0.
Except that setting wantobjects to 0 no longer affects the result.
Add a new parameter return_ints: specifying return_ints=True makes
Text.count() always returning the single count as an integer
instead of a 1-tuple or None.
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add optional 'filter' parameter to iterdump() that allows a "LIKE"
pattern for filtering database objects to dump.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Change the somewhat vague "listed below" to "listed in this chapter" in Doc/library/exceptions.rst.
The exceptions are listed in multiple sections after two intermediate sections.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* When called with a single argument to get a value, it allow to omit
the minus prefix.
* It can be called with keyword arguments to set attributes.
* w.wm_attributes(return_python_dict=True) returns a dict instead of
a tuple (it will be the default in future).
* Setting wantobjects to 0 no longer affects the result.
Update documentation with `__new__` and `__init__` entries.
Support use of `auto()` in tuple subclasses on member assignment lines. Previously, auto() was only supported on the member definition line either solo or as part of a tuple:
RED = auto()
BLUE = auto(), 'azul'
However, since Python itself supports using tuple subclasses where tuples are expected, e.g.:
from collections import namedtuple
T = namedtuple('T', 'first second third')
def test(one, two, three):
print(one, two, three)
test(*T(4, 5, 6))
# 4 5 6
it made sense to also support tuple subclasses in enum definitions.
The new `PyList_GetItemRef` is similar to `PyList_GetItem`, but returns
a strong reference instead of a borrowed reference. Additionally, if the
passed "list" object is not a list, the function sets a `TypeError`
instead of calling `PyErr_BadInternalCall()`.
---------
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Improve misleading TCP server docs and example.
socket.recv(), as documented by the Python reference documentation,
returns at most `bufsize` bytes, and the underlying TCP protocol means
there is no guaranteed correspondence between what is sent by the client
and what is received by the server.
This conflation could mislead readers into thinking that TCP is
datagram-based or has similar semantics, which will likely appear to
work for simple cases, but introduce difficult to reproduce bugs.
The text clearly seems to be referencing `TestFuncAcceptsSequencesMixin`,
for which no target is available. Name the class properly and suppress
the dangling reference.
Return files and directories from `pathlib.Path.glob()` if the pattern ends
with `**`. This is more compatible with `PurePath.full_match()` and with
other glob implementations such as bash and `glob.glob()`. Users can add a
trailing slash to match only directories.
In my previous patch I added a `FutureWarning` with the intention of fixing
this in Python 3.15. Upon further reflection I think this was an
unnecessarily cautious remedy to a clear bug.
Signed-off-by: Soumendra Ganguly <soumendraganguly@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
nedbat took issue with the phrasing "real module". I'm actually fine
with that phrasing, but I do think the `__future__` page should be clear
about the way in which the `__future__` module is special. (Yes, there
was a footnote linking to the future statements part of the reference,
but there should be upfront discussion).
I'm sympathetic to nedbat's claim that no one really cares about
`__future__._Feature`, so I've moved the interesting table up to the
top.
Add `ntpath.isreserved()`, which identifies reserved pathnames such as "NUL", "AUX" and "CON".
Deprecate `pathlib.PurePath.is_reserved()`.
---------
Co-authored-by: Eryk Sun <eryksun@gmail.com>
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
- consistently use correct parameter markup
- consistently use submodule name as database name
- improve accuracy of the dbm.dumb.open() spec
- remove dumbdbm class refs and replace them with generic "database object"
- use parameter list for dbm.dumb.open()
- add abbreviation directives for NDBM and GDBM
- consistently spell NDBM as NDBM
- silence broken ndbm class refs
- improve accuracy of dbm.ndbm.open() spec
- use replacement text for NDBM/GDBM file format incompatibility note
- add refs to other parts of the docs (dict, bytes, etc.)
- clarify whichdb() return value by using list markup
- silence refs to example or generic submodule methods (keys, get, etc.)
In 49f90ba we added support for the recursive wildcard `**` in
`pathlib.PurePath.match()`. This should allow arbitrary prefix and suffix
matching, like `p.match('foo/**')` or `p.match('**/foo')`, but there's a
problem: for relative patterns only, `match()` implicitly inserts a `**`
token on the left hand side, causing all patterns to match from the right.
As a result, it's impossible to match relative patterns from the left:
`PurePath('foo/bar').match('bar/**')` is true!
This commit reverts the changes to `match()`, and instead adds a new
`full_match()` method that:
- Allows empty patterns
- Supports the recursive wildcard `**`
- Matches the *entire* path when given a relative pattern
`threading.Lock` is now the underlying class and is constructable rather than the old
factory function. This allows for type annotations to refer to it which had no non-ugly
way to be expressed prior to this.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Use rst substitutions to reduce raw text duplication.
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Remove a double negative in the documentation of `mkdir()`'s *exist_ok*
parameter.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
The terminal CR -> NL mapping setting should be inherited in cbreak mode as OSes do not specify altering it as part of their stty cbreak mode definition.
The GH-93000 change set inadvertently caused a sentence in re.compile()
documentation to refer to details that no longer followed. Correct this
with a link to the Flags sub-subsection.
Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>
Allow `os.PathLike` objects to be passed as patterns to `pathlib.Path.glob()` and `rglob()`. (It's already possible to use them in `PurePath.match()`)
While we're in the area:
- Allow empty glob patterns in `PathBase` (but not `Path`)
- Speed up globbing in `PathBase` by generating paths with trailing slashes only as a final step, rather than for every intermediate directory.
- Simplify and speed up handling of rare patterns involving both `**` and `..` segments.
Tkinter is a fact, not necessarily a feature.
Reorganize editor key bindings in a logical order
and remove those that do not work, at least on Windows.
Improve shell bindings list.
This is a very soft deprecation of `PurePath.as_uri()`. We instead document
it as a `Path` method, and add a couple of sentences mentioning that it's
also available in `PurePath`.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
- Align the argument spec for fnmatch functions with the actual
implementation.
- Update Sphinx markup to recent recommandations.
- Add link to 'iterable' glossary entry.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Align the multiprocessing shared memory docs with Diatáxis's
recommendations for references.
- use a parameter list for the SharedMemory.__init__() argument spec
- use the imperative mode
- use versionadded, not versionchanged, for added parameters
- reflow touched lines according to SemBr
- add :class: and :mod: markups where needed
- fix incorrect escaping of a star in ShareableList arg spec
- mark up parameters with stars: *val*
- mark up list of built-in types using list markup
- remove unneeded parentheses from :meth: markups
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
`PyComplex_RealAsDouble()`/`PyComplex_ImagAsDouble` now try to convert
an object to a `complex` instance using its `__complex__()` method
before falling back to the ``__float__()`` method.
PyComplex_ImagAsDouble() also will not silently return 0.0 for
non-complex types anymore. Instead we try to call PyFloat_AsDouble()
and return 0.0 only if this call is successful.
On Windows, `os.path.isabs()` now returns `False` when given a path that
starts with exactly one (back)slash. This is more compatible with other
functions in `os.path`, and with Microsoft's own documentation.
Also adjust `pathlib.PureWindowsPath.is_absolute()` to call
`ntpath.isabs()`, which corrects its handling of partial UNC/device paths
like `//foo`.
Co-authored-by: Jon Foster <jon@jon-foster.co.uk>
The MagicMock documentation mentions magic methods several times without
actually pointing to the term in the glossary. This can be helpful for
people to fully understand what those magic methods are.
Make zipfile.ZipInfo.compress_level public.
A property is used to retain the behavior of the ._compresslevel.
People constructing zipfile.ZipInfo instances to pass into existing APIs to control per-file compression levels already treat this as public, there was never a reason for it not to be.
I used the more modern name compress_level instead of compresslevel as the keyword argument on other ZipFile APIs is called to be consistent with compress_type and a general long term preference of not runningwordstogether without a separator in names.
On Windows, set _O_NOINHERIT flag on file descriptors
created by os.pipe() and io.WindowsConsoleIO.
Add test_pipe_spawnl() to test_os.
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Previously the C implementation of pickle.Pickler and pickle.Unpickler
classes did not have such methods and they could only be used if
they were overloaded in subclasses or set as instance attributes.
Fixed calling super().persistent_id() and super().persistent_load() in
subclasses of the C implementation of pickle.Pickler and pickle.Unpickler
classes. It no longer causes an infinite recursion.
It can be used to set the location of a .python_history file
---------
Co-authored-by: Levi Sabah <0xl3vi@gmail.com>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* gh-113536: Expose `os.waitid` on macOS
This API has been available on macOS for a long time, but was
explicitly excluded due to unspecified problems with the API
in ancient versions of macOS.
* Document that the API is available on macOS starting in Python 3.13
Sebastian Pipping
Shantanu
Carl Bordum Hansen
Erlend E. Aasland
Vinay Sajip
Raymond Hettinger
Ethan Furman
Kerim Kabirov
Jan Max Meyer
Jelle Zijlstra
Tahoma Software
Phil Elson
Antti Haapala
Hugo van Kemenade
Terry Jan Reedy
Barney Gale
bssyousefi
Skip Montanaro
Matan Perelman
Laurie O
mauricelambert
Oh seungmin
Adorilson Bezerra
partev
jmcb
Stanley
Pablo Galindo Salgado
NewUserHa
Gregory P. Smith
Bogdan Romanyuk
Victor Westerhuis
Eugene Toder
talcs
Naglis Jonaitis
Serhiy Storchaka
Brian Schubert
Daniel Haag
0xflotus
Masayuki Moriyama
Simon A. Eugster
Daler
Sebastian Rittau
Thomas Grainger
Matthew Hughes
Furkan Onder
Rami
Stevoisiak
Jamie Phan
Sam Gross
monkeyman192
Martijn Pieters
David Hewitt
Ned Batchelder
Victor Stinner
Stanislav Lyu
Alex Waygood
Christophe Papazian
qqwqqw689
Steve Dower
Petr Viktorin
Nikita Sobolev
Brandt Bucher
Mike Zimin
Brett Cannon
Mark Shannon
Edgar Ramírez Mondragón
Finite State Machine
da-woods
Mariusz Felisiak
HarryLHW
Dai Wentao
Kristján Valur Jónsson
AN Long
John Belmonte
patenaud
Steven Ward
Justin Williams
Michal Kaptur
Oleg Iarygin
Nicholas Hollander
Guido van Rossum
Christophe Nanteuil
technillogue
srinivasan
Pradyot Ranjan
Aidan Holm
Bradley Reynolds
Soumendra Ganguly
Bhushan Mohanraj
Sergey B Kirpichev
plokmijnuhby
Daniel Hollas
Ville Skyttä
Géry Ogam
Kirill Podoprigora
cdzhan
Adam Turner
David H. Gutteridge
DerSchinken
Miyashita Yosuke
Karolina Surma
Steffen Zeile
Kuan-Wei Chiu
solya0x
Zackery Spytz
Ronald Oussoren
InSync
Pierre Equoy
Andrew Zipperer
Alois Klink
Stefano Rivera
William Andrea
Hugo van Kemenade
Rodrigo Girão Serrão
Ege Akman
John D. McDonald
Parth Doshi
Irit Katriel
Delgan
Samet YASLAN
John Hawkinson
Inada Naoki