Commit Graph

450 Commits

Author SHA1 Message Date
Victor Stinner 424049cc11
gh-105145: Remove old functions to config Python init (#105154)
Remove the following old functions to configure the Python
initialization, deprecated in Python 3.11:

* PySys_AddWarnOptionUnicode()
* PySys_AddWarnOption()
* PySys_AddXOption()
* PySys_HasWarnOptions()
* PySys_SetArgvEx()
* PySys_SetArgv()
* PySys_SetPath()
* Py_SetPath()
* Py_SetProgramName()
* Py_SetPythonHome()
* Py_SetStandardStreamEncoding()
* _Py_SetProgramFullPath()

Most of these functions are kept in the stable ABI, except:

* Py_SetStandardStreamEncoding()
* _Py_SetProgramFullPath()

Update Doc/extending/embedding.rst and Doc/extending/extending.rst to
use the new PyConfig API.

_testembed.c:

* check_stdio_details() now sets stdio_encoding and stdio_errors
  of PyConfig.
* Add definitions of functions removed from the API but kept in the
  stable ABI.
* test_init_from_config() and test_init_read_set() now use
  PyConfig_SetString() instead of PyConfig_SetBytesString().

Remove _Py_ClearStandardStreamEncoding() internal function.
2023-06-01 09:14:02 +02:00
Gregory Szorc 5360cb3d56
gh-101282: Apply BOLT optimizations to libpython for shared builds (#104709)
Apply BOLT optimizations to libpython for shared builds. Most of the C
code is in libpython so it is critical to apply BOLT there fully realize
BOLT benefits.

This change also reworks how BOLT instrumentation is applied. It
effectively removes the readelf based logic added in gh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform BOLT
optimizations without having to manually delete the output binary to
force a new bolt run.

Also:
- add a clean-bolt target for purging BOLT files and hook that up to the
  clean target
- .gitignore BOLT related files

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need to
`rm profile-bolt-stamp` to force a BOLT run because there aren't
appropriate non-phony targets for BOLT's make target to depend on.

To make it easier to iterate on custom BOLT settings, the flags to pass
to instrumentation and application are now defined in configure and can
be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
2023-05-22 13:45:20 +02:00
partev 06eeee97e3
Replace "OS X" with "macOS" (#104653) 2023-05-20 01:25:52 -07:00
Rafael Fontenelle 68ed2a2a3f
GH-103484: Fix redirected permanently URLs (#104001)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
2023-05-02 09:34:44 +03:00
partev c9134fb228
DOC: remove reference to OpenSolaris (#102671) 2023-04-25 10:45:18 +00:00
Christopher Chavez 014f131934
Change 'dependant' to 'dependent' (#103745)
The word 'dependent' is both an adjective and a noun. A 'dependant' is a British alternative spelling for the noun form. In idlelib.sidebar, 'OS-dependant' is an adjective and clearly wrong. In 'Using', 'dependant' as a noun would be acceptable in Britain, but we use American spellings in Python docs.
https://www.merriam-webster.com/words-at-play/spelling-variants-dependent-vs-dependant
2023-04-24 09:40:30 -04:00
Pradyun Gedam ece20dba12
gh-95299: Stop installing setuptools as a part of ensurepip and venv (#101039)
Remove the bundled setuptools wheel from ensurepip, and stop installing setuptools in environments created by venv.

Co-Authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
2023-04-17 23:43:34 -05:00
Furkan Onder d65ed693a8
GH-83893: Cross reference env. vars and -X command line options (GH-103414)
Co-authored-by: Erlend E. Aasland
2023-04-12 01:44:35 -07:00
Julien Palard 61b9ff35cb
gh-101100: Documenting --prefix and --exec-prefix. (GH-102695)
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-03-15 16:10:03 +01:00
Paul Watson 80abd62647
gh-102354: change python3 to python in docs examples (#102696) 2023-03-14 12:40:12 -06:00
Matthias Görgens eff9f43924
gh-96821: Add config option `--with-strict-overflow` (#96823)
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Shantanu <hauntsaninja@gmail.com>
2023-03-04 13:39:52 -08:00
Steve Dower eda60916bc
gh-101467: Correct py.exe handling of prefix matches and cases when only one runtime is installed (GH-101468) 2023-02-01 21:06:56 +00:00
socal-nerdtastic 666c0840dc
Fix trivial typo in shebang example (GH-101385)
The example was showing the current version, but should be pinned to 3.7 to match the example command.
2023-01-28 15:18:04 +02:00
Steve Dower 30753c37c6
gh-100247: Improve documentation for custom shebang commands in py.exe launcher (GH-101083) 2023-01-16 17:01:04 +00:00
Steve Dower 468c3bf798
gh-100247: Fix py.exe launcher not using entire shebang command for finding custom commands (GH-100944) 2023-01-13 11:49:01 +00:00
Zachary Ware 25590eb5de
gh-100540: Remove obsolete '--with-system-ffi' configure option (GH-100544)
It has had no effect on non-macOS platforms for a long time, and has had
the non-obvious effect of invoking `pkg_config` and not setting
`-DUSING_APPLE_OS_LIBFFI` on macOS since GH-22855.
2022-12-28 13:36:03 -06:00
Victor Stinner 5837e5f347
gh-100008: VS 2017 is required since Python 3.11 (#100045) 2022-12-06 14:35:32 +01:00
Victor Stinner 5ea052bb0c
gh-100008: Document Python build requirements (#100009)
Document also configure --without-freelists option added to Python
3.11.
2022-12-05 15:41:44 +01:00
Victor Stinner 9a7e9f9921
gh-99289: Add COMPILEALL_OPTS to Makefile (#99291)
Add COMPILEALL_OPTS variable in Makefile to override compileall
options (default: -j0) in "make install". Also merge the compileall
commands into a single command building PYC files for the all
optimization levels (0, 1, 2) at once.

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2022-11-14 13:43:45 +01:00
Michael Droettboom 2844aa6a8e
Support comparing two sets of pystats (GH-98816)
This adds support for comparing pystats collected from two different builds.

- The `--json-output` can be used to load in a set of raw stats and output a
  JSON file.
- Two of these JSON files can be provided on the next run, and then comparative
  results between the two are output.
2022-11-04 10:15:54 +00:00
Steve Dower 88297e2a8a
gh-98692: Enable treating shebang lines as executables in py.exe launcher (GH-98732) 2022-10-31 21:05:50 +00:00
Erlend E. Aasland 723ebe76e7
gh-96143: Improve perf profiler docs (#96445) 2022-10-27 14:06:48 +01:00
Stanley 268129a74f
docs: Change links to label refs (#98454)
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-10-25 20:26:28 -07:00
Phil Elson 1a6bacb31f
[doc] Refresh the venv introduction documentation, and correct the statement about VIRTUAL_ENV (GH-98350)
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-10-19 15:49:34 +01:00
Hugo van Kemenade fa2d43e518
Docs: Fix backtick errors found by sphinx-lint (#97998)
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
2022-10-06 18:01:30 -07:00
Xiao Chen 4875433682
Remove extra spaces in custom openSSL documentation. (#93568) 2022-10-06 17:42:08 -07:00
Ben Faulhaber a120b9f25d
gh-97754: Update doc for default location of per-user installs on Windows (GH-97756) 2022-10-04 17:04:41 +01:00
Serhiy Storchaka dd53b79de0
gh-96959: Update more HTTP links (GH-97536)
Use HTTPS for documents which are available by both HTTP and HTTPS
links, but there is no redirection from HTTP to HTTPS or vice versa.
2022-09-27 14:08:11 +03:00
Gregory P. Smith 34de67c094
gh-96512: Update int_max_str docs to say 3.11 (#96942)
It was unknown if it'd be before 3.11.0 when creating the original
changes.  It's in 3.11rc2, so 3.11 it is.
2022-09-19 16:43:11 -07:00
Dong-hee Na e47b96c44f
gh-89536: Use ThinLTO policy if possible (gh-96766) 2022-09-16 19:40:05 +09:00
Gregory P. Smith 511ca94520
gh-95778: CVE-2020-10735: Prevent DoS by very large int() (#96499)
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.

This PR comes fresh from a pile of work done in our private PSRT security response team repo.

Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>
Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).

<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->

I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#). Much of that text wound up in the Issue. Backports PRs already exist. See the issue for links.
2022-09-02 09:35:08 -07:00
Pablo Galindo Salgado f49dd54b72
gh-96143: Add some comments and minor fixes missed in the original PR (#96433)
* gh-96132: Add some comments and minor fixes missed in the original PR

* Update Doc/using/cmdline.rst

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-08-30 19:37:22 +01:00
Pablo Galindo Salgado 6d791a9736
gh-96143: Allow Linux perf profiler to see Python calls (GH-96123)
⚠️  ⚠️ Note for reviewers, hackers and fellow systems/low-level/compiler engineers ⚠️ ⚠️ 

If you have a lot of experience with this kind of shenanigans and want to improve the **first** version, **please make a PR against my branch** or **reach out by email** or **suggest code changes directly on GitHub**. 

If you have any **refinements or optimizations** please, wait until the first version is merged before starting hacking or proposing those so we can keep this PR productive.
2022-08-30 10:11:18 -07:00
Dong-hee Na 6ec57e7c5a
gh-90536: Fix link syntax to LLVM-BOLT repository (gh-96141) 2022-08-21 00:11:35 +09:00
Kevin Modzelewski 214eb2cce5
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer

Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.

It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).

Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.

The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).

The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.

This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.

* Simplify the build flags

* Add a NEWS entry

* Update Makefile.pre.in

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Update configure.ac

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Add myself to ACKS

* Add docs

* Other review comments

* fix tab/space issue

* Make it more clear that --enable-bolt is experimental

* Add link to bolt's github page

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-19 06:33:54 +09:00
Howie Zhao d25ff1f613
gh-95155: Update "Using Python on a Mac" documentation (GH-95284)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2022-08-06 18:50:28 +02:00
Victor Stinner 398ed84dc4
gh-93744: Remove configure --with-cxx-main option (#95651)
Remove the "configure --with-cxx-main" build option: it didn't work
for many years. Remove the MAINCC variable from configure and
Makefile.

The MAINCC variable was added by the issue gh-42471: commit
0f48d98b74. Previously, --with-cxx-main
was named --with-cxx.

Keep CXX and LDCXXSHARED variables, even if they are no longer used
by Python build system.
2022-08-05 13:26:58 +02:00
fluesvamp c47d09a4ac
Remove unnecessary text from documentation. (GH-95670) 2022-08-04 21:18:05 +01:00
Serhiy Storchaka f79547a429
gh-91838: Use HTTPS links in docs for resources which redirect to HTTPS (GH-95527)
If an HTTP link is redirected to a same looking HTTPS link, the latter can
be used directly without changes in readability and behavior.
It protects from a men-in-the-middle attack.

This change does not affect Python examples.
2022-08-04 10:13:49 +03:00
Steve Dower 67840edb28
gh-94399: Restore PATH search behaviour of py.exe launcher for '/usr/bin/env' shebang lines (GH-95582) 2022-08-03 22:18:51 +01:00
Steve Dower d2c1a9c76c
gh-92219: Clarify that some options to the installer may break the install (GH-95548) 2022-08-02 00:57:46 +01:00
Christian Heimes f81a6c5fc7
gh-95415: Make availability directive consistent (GH-95416) 2022-07-29 16:42:09 +02:00
Christian Heimes a566912049
gh-85454: Remove distutils documentation (#95239)
Most places now refer to setuptools or link to setuptools documentation.
Some examples like zipapp need to be updated later.
2022-07-25 15:50:46 +02:00
Anthony Shaw b1924b1006
GH-95029: Describe Windows Store package isolation and redirection in more detail (GH-95030) 2022-07-21 01:01:23 +01:00
Ned Batchelder 3440d197a5
Docs: remove redundant "adverb-adjective" hyphens from compound modifiers (GH-94551)
Discussion: https://discuss.python.org/t/slight-grammar-fix-throughout-adverbs-dont-need-hyphen/17021
2022-07-05 11:16:10 +02:00
Oleg Iarygin f62ff97f31
gh-93851: Fix all broken links in Doc/ (GH-93853) 2022-06-21 20:55:18 +02:00
Matthias Köppe 61f24e7885
gh-90539: doc: Expand on what should not go into CFLAGS, LDFLAGS (#92754) 2022-06-20 15:58:02 +02:00
Éric 8aa9d40b00
gh-90300: split --help output into separate options (#30331)
Make --help output shorter and add new help options.

--help-env, --help-xoptions and --help-all command-line options are
added to complement --help.
2022-06-01 05:50:01 -04:00
Victor Stinner c7667a2d35
gh-93103: Enhance PyConfig.parser_debug documentation (#93186)
Document that -d option and PYTHONDEBUG env var require a debug
build of Python. Also mention them in the debug build documentation.
2022-05-24 23:23:01 +02:00
CAM Gerlach f1bbcba74f
gh-76773: Update docs mentioning no-longer-supported Windows versions & features (GH-92529) 2022-05-10 09:30:32 +03:00