Functions re.sub() and re.subn() and corresponding re.Pattern methods
are now 2-3 times faster for replacement strings containing group references.
Closes#91524
Primarily authored by serhiy-storchaka Serhiy Storchaka
Minor-cleanups-by: Gregory P. Smith [Google] <greg@krypto.org>
Rely on the title of the linked internal page instead of putting the title. Sphinx will render with the title correctly, and this will reduce work for translators
* Link ZipFile in What's New entry discussing it
* Add entry for new ZipFile.mkdir method
* Add entry for new zipfile.Path.stem/suffix/suffixes methods
* Add missing line breaks between zipfile bullet list items
* Add entry for new logging.getLevelNamesMapping function
* Add entry for SysLogHandler.createSocket to whatsnew
* Add missing line break between logging bullet list items
The os module and the PyUnicode_FSDecoder() function no longer accept
bytes-like paths, like bytearray and memoryview types: only the exact
bytes type is accepted for bytes strings.
* Whatsnew: Convert literals in enum section to actual x-references
* Whatsnew: Rewrite enum section for clear and consistant phrasing
* Whatsnew: Combine directly related enum items instead of seperating them
* gh-98250: Describe __str__/__format__ changes more clearly/accurately
* Tweak enum section language per feedback from Ethan
* Add line breaks & ref targets to Whatsnew to prepare for future changes
* Use standard heading underbar symbols for H4 sections
* Flatten Porting subsection; clarify scope of/link Python->CAPI sections
* Move C API pending deprecations to C API section, to match the others
Part of #95913
Forward port of #93306, which was a backport of #93185, to address #84694
This adds the What's New entry for the removal of the subinterpreter-related env variable, build-time flag, etc. As @ericsnowcurrently was author of the original changes, I added him as a co-author to the commit.
This addition to the Python 3.11 What's New document were only made to the Python 3.11 branch during the backport process, and not added to the version in `main`. Forward-porting it ensures the docs retain these additions for the future, rather than being lost in a legacy Python versions, allows it to be be edited as part of #95913 , and avoids merge conflicts with routine back-ports of PRs touching it.
I've pulled in the addition exactly as-is with no modifications; any editing will be done in future PRs (and therefore can be reviewed and backported accordingly).
The one other such addition is forward-ported in #98344
This is the next step for deprecating child watchers.
Until we've removed the API completely we have to use it, so this PR is mostly suppressing a lot of warnings when using the API internally.
Once the child watcher API is totally removed, the two child watcher implementations we actually use and need (Pidfd and Thread) will be turned into internal helpers.
* Add/refine cross references to items in other lang changes section
* Unify context manager exception changes into single non-repetitive item
* More clearly describe the intent and consequences of the -P option
* Apply minor clarifications & copyedits to rest of section
* Tweak the formatting of module references
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
* Move Windows py.exe improvements from Typing section to New Features
* Add ref target label and use literal for py.exe
* Be clearer/explict about what legacy version arg components reprisent
* Apply other minor clarity and textual fixes to py.exe launcher text
* Refine phrasing of legacy sentence of py.exe desc
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Remove the Tools/demo/ directory which contained old demo scripts. A
copy can be found in the old-demos project:
https://github.com/gvanrossum/old-demos
Remove the following old demo scripts:
* beer.py
* eiffel.py
* hanoi.py
* life.py
* markov.py
* mcast.py
* queens.py
* redemo.py
* rpython.py
* rpythond.py
* sortvisu.py
* spreadsheet.py
* vector.py
Changes:
* Remove a reference to the redemo.py script in the regex howto
documentation.
* Remove a reference to the removed Tools/demo/ directory in the
curses documentation.
* Update PC/layout/ to remove the reference to Tools/demo/ directory.
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.
This PR comes fresh from a pile of work done in our private PSRT security response team repo.
Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>
Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).
<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->
I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#). Much of that text wound up in the Issue. Backports PRs already exist. See the issue for links.
The previous wording of this entry suggests that CPython
won't work if optional compiler features are enabled.
That's not the case. The change is that we require C11 rather
than C89.
Note that PEP 7 does say "Python 3.11 and newer versions use C11
without optional features." It is correct there: that's
not a guide for users who compile Python, but for CPython devs
who must avoid the features.
* Add support for the BOLT post-link binary optimizer
Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.
It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).
Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.
The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).
The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.
This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.
* Simplify the build flags
* Add a NEWS entry
* Update Makefile.pre.in
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Update configure.ac
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Add myself to ACKS
* Add docs
* Other review comments
* fix tab/space issue
* Make it more clear that --enable-bolt is experimental
* Add link to bolt's github page
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Treat tp_weakref and tp_dictoffset like other opaque slots for multiple inheritance.
* Document Py_TPFLAGS_MANAGED_DICT and Py_TPFLAGS_MANAGED_WEAKREF in what's new.
An unrecognized format character in PyUnicode_FromFormat() and
PyUnicode_FromFormatV() now sets a SystemError.
In previous versions it caused all the rest of the format string to be
copied as-is to the result string, and any extra arguments discarded.
gh-93243
This PR is required to reduce diffs of the following porting (no need to either maintain documentation and tests consistent with each porting step, or try to port everything and remove smtpd in a single PR).
Automerge-Triggered-By: GH:warsaw
If an HTTP link is redirected to a same looking HTTPS link, the latter can
be used directly without changes in readability and behavior.
It protects from a men-in-the-middle attack.
This change does not affect Python examples.
* Store tp_weaklist on the interpreter state for static builtin types.
* Factor out _PyStaticType_GET_WEAKREFS_LISTPTR().
* Add _PyStaticType_ClearWeakRefs().
* Add a comment about how _PyStaticType_ClearWeakRefs() loops.
* Document the change.
* Update Doc/whatsnew/3.12.rst
* Fix a typo.
These headings were at the same level as the "Deprecated" heading, but
likely intended to be a subheading within that section.
Co-authored-by: Pradyun Gedam <pradyunsg@users.noreply.github.com>
* gh-93883: elide traceback indicators when possible
Elide traceback column indicators when the entire line of the
frame is implicated. This reduces traceback length and draws
even more attention to the remaining (very relevant) indicators.
Example:
```
Traceback (most recent call last):
File "query.py", line 99, in <module>
bar()
File "query.py", line 66, in bar
foo()
File "query.py", line 37, in foo
magic_arithmetic('foo')
File "query.py", line 18, in magic_arithmetic
return add_counts(x) / 25
^^^^^^^^^^^^^
File "query.py", line 24, in add_counts
return 25 + query_user(user1) + query_user(user2)
^^^^^^^^^^^^^^^^^
File "query.py", line 32, in query_user
return 1 + query_count(db, response['a']['b']['c']['user'], retry=True)
~~~~~~~~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable
```
Rather than going out of our way to provide indicator coverage
in every traceback test suite, the indicator test suite should
be responible for sufficient coverage (e.g. by adding a basic
exception group test to ensure that margin strings are covered).
Remove the ssl.wrap_socket() function, deprecated in Python 3.7:
instead, create a ssl.SSLContext object and call its
sl.SSLContext.wrap_socket() method. Any package that still uses
ssl.wrap_socket() is broken and insecure. The function neither sends
a SNI TLS extension nor validates server hostname. Code is subject to
CWE-295 : Improper Certificate Validation.
zipimport: Remove find_loader() and find_module() methods, deprecated
in Python 3.10: use the find_spec() method instead. See PEP 451 for
the rationale.
xml.etree: Remove the ElementTree.Element.copy() method of the pure
Python implementation, deprecated in Python 3.10, use the copy.copy()
function instead. The C implementation of xml.etree has no copy()
method, only a __copy__() method.
Remove the pure Python implementation of hashlib.pbkdf2_hmac(),
deprecated in Python 3.10. Python 3.10 and newer requires OpenSSL
1.1.1 or newer (PEP 644), this OpenSSL version provides a C
implementation of pbkdf2_hmac() which is faster.
Remove the locale.format() function, deprecated in Python
3.7: use locale.format_string() instead.
Remove TestFormatPatternArg test case: it is irrelevant for
locale.format_string() which accepts complex formats.
gzip: Remove the filename attribute of gzip.GzipFile,
deprecated since Python 2.6, use the name attribute instead. In write
mode, the filename attribute added '.gz' file extension if it was not
present.
Remove io.OpenWrapper and _pyio.OpenWrapper, deprecated in Python
3.10: just use :func:`open` instead. The open() (io.open()) function
is a built-in function. Since Python 3.10, _pyio.open() is also a
static method.
Move the follow functions and type from frameobject.h to pyframe.h,
so the standard <Python.h> provide frame getter functions:
* PyFrame_Check()
* PyFrame_GetBack()
* PyFrame_GetBuiltins()
* PyFrame_GetGenerator()
* PyFrame_GetGlobals()
* PyFrame_GetLasti()
* PyFrame_GetLocals()
* PyFrame_Type
Remove #include "frameobject.h" from many C files. It's no longer
needed.
* What's new in 3.10: fix link to issue
* What's new in 3.10: fix link to GH issue
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Deprecate global configuration variable like
Py_IgnoreEnvironmentFlag: the Py_InitializeFromConfig() API should be
instead.
Fix declaration of Py_GETENV(): use PyAPI_FUNC(), not PyAPI_DATA().
This was added for bpo-40514 (gh-84694) to test out a per-interpreter GIL. However, it has since proven unnecessary to keep the experiment in the repo. (It can be done as a branch in a fork like normal.) So here we are removing:
* the configure option
* the macro
* the code enabled by the macro
Added a new stable API function ``PyType_FromMetaclass``, which mirrors
the behavior of ``PyType_FromModuleAndSpec`` except that it takes an
additional metaclass argument. This is, e.g., useful for language
binding tools that need to store additional information in the type
object.
The following sqlite3 features were deprecated in 3.10, scheduled for
removal in 3.12:
- sqlite3.OptimizedUnicode (gh-23163)
- sqlite3.enable_shared_cache (gh-24008)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Signed-off-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Remove the PyUnicode_InternImmortal() function and the
SSTATE_INTERNED_IMMORTAL macro.
The PyUnicode_InternImmortal() function is still exported in the
stable ABI. The function is removed from the API.
PyASCIIObject.state.interned size is now a single bit, rather than 2
bits.
Keep SSTATE_NOT_INTERNED and SSTATE_INTERNED_MORTAL macros for
backward compatibility, but no longer use them internally since the
interned member is now a single bit and so can only have two values
(interned or not interned).
Update stats of _PyUnicode_ClearInterned().
Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.
Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.
Rename functions:
* PyToken_OneChar() => _PyToken_OneChar()
* PyToken_TwoChars() => _PyToken_TwoChars()
* PyToken_ThreeChars() => _PyToken_ThreeChars()
I merged this without a What's New entry to avoid merge conflicts, so here's the follow-up adding the entry.
@Kab1r do you mind reviewing?
Closes#91928
Automerge-Triggered-By: GH:pganssle
We could try to remedy this by taking a slice, but we then run into an issue where the empty string will match altsep on POSIX. That rabbit hole could keep getting deeper.
A proper fix for the original issue involves making pathlib's path normalisation more configurable - in this case we want to retain trailing slashes, but in other we might want to preserve `./` prefixes, or elide `../` segments when we're sure we won't encounter symlinks.
This reverts commit ea2f5bcda1.
Only sequence of ASCII digits is now accepted as a numerical reference.
The group name in bytes patterns and replacement strings can now only
contain ASCII letters and digits and underscore.
Add methods enterContext() and enterClassContext() in TestCase.
Add method enterAsyncContext() in IsolatedAsyncioTestCase.
Add function enterModuleContext().
* gh-92308: Add Pending Removal section to 3.11 What's New
* Use compact list; drop attributions
* Add short text, and also missing PyUnicode_InternImmortal
* Fix formatting
* markup fix
* Update Doc/whatsnew/3.11.rst
Co-authored-by: Victor Stinner <vstinner@python.org>
* Apply suggestions from code review
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
Add the -P command line option and the PYTHONSAFEPATH environment
variable to not prepend a potentially unsafe path to sys.path.
* Add sys.flags.safe_path flag.
* Add PyConfig.safe_path member.
* Programs/_bootstrap_python.c uses config.safe_path=0.
* Update subprocess._optim_args_from_interpreter_flags() to handle
the -P command line option.
* Modules/getpath.py sets safe_path to 1 if a "._pth" file is
present.
Only sequence of ASCII digits will be accepted as a numerical reference.
The group name in bytes patterns and replacement strings could only
contain ASCII letters and digits and underscore.
This gets all the major items in #91491. However, I didn't get around to
adding what's new entries for the large clump of changes in the last
bullet point in the issue.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* Add a new :gh:`...` role for GitHub issues.
* Fix a GitHub id to use the :gh: role.
* Add Misc/NEWS entry.
* Refactoring and rephrasing.
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Also inline necessary functionality from `sndhdr` into `email.mime.audio` for `MIMEAudio`.
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
The existing event loop `start_tls()` method is not sufficient for
connections using the streams API. The existing StreamReader works
because the new transport passes received data to the original protocol.
The StreamWriter must then write data to the new transport, and the
StreamReaderProtocol must be updated to close the new transport
correctly.
The new StreamWriter `start_tls()` updates itself and the reader
protocol to the new SSL transport.
Co-authored-by: Ian Good <icgood@gmail.com>
* Deprecate imghdr
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Update Doc/whatsnew/3.11.rst
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Inline `imghdr` into `email.mime.image`
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Barry Warsaw <barry@python.org>
Remove the Include/code.h header file. C extensions should only
include the main <Python.h> header file.
Python.h includes directly Include/cpython/code.h instead.
Copying and pickling instances of subclasses of builtin types
bytearray, set, frozenset, collections.OrderedDict, collections.deque,
weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes
implemented as slots.
* Revert "bpo-46850: Move _PyInterpreterState_SetEvalFrameFunc() to internal C API (GH-32054)"
This reverts commit f877b40e3f.
* Revert "bpo-46850: Move _PyEval_EvalFrameDefault() to internal C API (GH-32052)"
This reverts commit b9a5522dd9.
Serhiy Storchaka
C.A.M. Gerlach
Carl Meyer
Rafael Fontenelle
Victor Stinner
Kumar Aditya
Julien Palard
Pablo Galindo Salgado
Hugo van Kemenade
Brett Cannon
Barry Warsaw
Adam Turner
180909
Ev2geny
Guido van Rossum
Will Hawkins
Ofey Chan
Stanley
Gregory P. Smith
Benjamin Peterson
Dong-hee Na
adphrost
Laurie O
Maksym Medvied
Steve Dower
Gregory P. Smith
Petr Viktorin
Kevin Modzelewski
Irit Katriel
Mark Shannon
Christian Heimes
Łukasz Langa
Stanislav Zmiev
esc
Oleg Iarygin
Brandt Bucher
Eric Snow
Ken Jin
Terry Jan Reedy
Ethan Furman
Erlend Egeberg Aasland
Zachary Ware
Frank Dana
MonadChains
Pradyun Gedam
John Belmonte
Ned Batchelder
KotlinIsland
Inada Naoki
Ken Jin
Samodya Abeysiriwardane
georgically
Wenzel Jakob
Miro Hrončok
Shantanu
Jelle Zijlstra
Duprat
Raymond Hettinger
Paul Ganssle
Barney Gale
Ezio Melotti
Erlend Egeberg Aasland
Alex Waygood
Dennis Sweeney
Eisuke Kawashima
Zac Hatfield-Dodds
Yurii Karabas
Sebastian Rittau
Brandt Bucher