Commit Graph

1092 Commits

Author SHA1 Message Date
Jeffery To 990cb3676c
gh-104692: Include commoninstall as a prerequisite for bininstall (#104693)
This ensures that `commoninstall` is completed before `bininstall` is
started when parallel builds are used (`make -j install`), and so the
`python3` symlink is only installed after all standard library modules
are installed.

Co-authored-by: Zachary Ware <zachary.ware@gmail.com>
2023-06-07 08:59:56 +02:00
Victor Stinner 0202aa002e
gh-102304: Fix up Simple ABI doc (GH-105351) 2023-06-06 13:03:51 +02:00
Victor Stinner bae415ad02
gh-102304: doc: Add links to Stable ABI and Limited C API (#105345)
* Add "limited-c-api" and "stable-api" references.
* Rename "stable-abi-list" reference to "limited-api-list".
* Makefile: Document files regenerated by "make regen-limited-abi"
* Remove first empty line in generated files:

  - Lib/test/test_stable_abi_ctypes.py
  - PC/python3dll.c
2023-06-06 08:40:32 +00:00
Eric Snow e6373c0d8b
gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-105258)
The _xxsubinterpreters module was meant to only use public API.  Some internal C-API usage snuck in over the last few years (e.g. gh-28969).  This fixes that.
2023-06-02 22:52:33 +00:00
Mark Shannon 4bfa01b9d9
GH-104584: Plugin optimizer API (GH-105100) 2023-06-02 11:46:18 +01:00
Erlend E. Aasland 8a8ebf2e3d
GH-89886: Bump to GNU Autoconf v2.71 (#104925)
Co-authored-by: Christian Heimes <christian@python.org>
2023-06-01 23:44:25 +02:00
Victor Stinner ec0082ca46
gh-105182: Remove PyEval_AcquireLock() and PyEval_InitThreads() (#105183)
Remove functions in the C API:

* PyEval_AcquireLock()
* PyEval_ReleaseLock()
* PyEval_InitThreads()
* PyEval_ThreadsInitialized()

But keep these functions in the stable ABI.

Mention "make regen-limited-abi" in "make regen-all".
2023-06-01 13:41:56 +02:00
Victor Stinner e399f46a77
gh-104773: PEP 594: Remove the imghdr module (#104777)
* Remove the Lib/test/imghdrdata/ directory.
* Copy 5 pictures (gif, png, ppm, pgm, xbm) from removed
  Lib/test/imghdrdata/ to a new Lib/test/tkinterdata/ directory.
* Update Sphinx from 4.5 to 6.2 in Doc/requirements.txt.
2023-05-26 13:29:45 +00:00
Victor Stinner ded5f1f287
gh-104773: PEP 594: Remove the nntplib module (#104894)
* socket_helper.transient_internet() no longer imports nntplib to
  catch nntplib.NNTPTemporaryError.
* ssltests.py no longer runs test_nntplib.
* "make quicktest" no longer runs test_nntplib.
* WASM: remove nntplib from OMIT_NETWORKING_FILES.
* Remove mentions to nntplib in the email documentation.
2023-05-24 21:15:43 +00:00
Jonathan Protzenko 160321e530
gh-99108: Refresh HACL* (#104808)
Refresh HACL* from upstream to improve SHA2 performance and fix a 32-bit issue in SHA3.
2023-05-24 13:30:11 -07:00
Victor Stinner 7b00940f69
gh-104773: PEP 594: Remove the sndhdr module (#104774)
Remove the Lib/test/sndhdrdata/ directory.
2023-05-24 09:32:18 +00:00
Victor Stinner ae00b810d1
gh-104780: Remove 2to3 program and lib2to3 module (#104781)
* Remove the Tools/scripts/2to3 script.
* Remove the Lib/test/test_lib2to3/ directory.
* Doc/tools/extensions/pyspecific.py: remove the "2to3fixer" object
  type.
* Makefile and PC/layout/main.py no longer compile lib2to3 grammar
  files.
* Update Makefile for 2to3 removal.
2023-05-23 19:40:02 +02:00
Gregory Szorc 5360cb3d56
gh-101282: Apply BOLT optimizations to libpython for shared builds (#104709)
Apply BOLT optimizations to libpython for shared builds. Most of the C
code is in libpython so it is critical to apply BOLT there fully realize
BOLT benefits.

This change also reworks how BOLT instrumentation is applied. It
effectively removes the readelf based logic added in gh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform BOLT
optimizations without having to manually delete the output binary to
force a new bolt run.

Also:
- add a clean-bolt target for purging BOLT files and hook that up to the
  clean target
- .gitignore BOLT related files

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need to
`rm profile-bolt-stamp` to force a BOLT run because there aren't
appropriate non-phony targets for BOLT's make target to depend on.

To make it easier to iterate on custom BOLT settings, the flags to pass
to instrumentation and application are now defined in configure and can
be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
2023-05-22 13:45:20 +02:00
Mark Shannon f7df173949
GH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (#104508) 2023-05-17 14:17:16 +01:00
Gregory Szorc 9084e1b04f
gh-104523: Inline minimal PGO rules (#104524)
Inline profiling rules where the existing indirection was unneeded.
2023-05-16 10:04:58 +02:00
Jelle Zijlstra 24d8b88420
gh-103763: Implement PEP 695 (#103764)
This implements PEP 695, Type Parameter Syntax. It adds support for:

- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules 

Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-15 20:36:23 -07:00
Gregory Szorc a6bcc8fb92
gh-104490: Consistently define phony make targets (#104491)
By convention make targets that don't refer to a file have a dependency
on the fake .PHONY target/file. This ensures that these targets are
always evaluated because there is no rule to create a .PHONY file
and that will force make to think the rule is out of date and needs
to be rebuilt.

This commit consistently associates virtual targets with .PHONY by
declaring the .PHONY dependency immediately above the make rule. This
should avoid race conditions and avoidable rebuilds across multiple make
invocations.
2023-05-15 08:48:34 +00:00
Jelle Zijlstra b7597dab2c
gh-86275: Fix install of new _hypothesis_stubs directory (#104425) 2023-05-12 15:51:46 +00:00
Jonathan Protzenko 15665d896b
gh-99108: Replace SHA3 implementation HACL* version (#103597)
Replaces our built-in SHA3 implementation with a verified one from the HACL* project.

This implementation is used when OpenSSL does not provide SHA3 or is not present.

3.11 shiped with a very slow tiny sha3 implementation to get off of the <=3.10 reference implementation that wound up having serious bugs. This brings us back to a reasonably performing built-in implementation consistent with what we've just replaced our other guaranteed available standard hash algorithms with: code from the HACL* project.

---------

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-05-07 20:50:04 -07:00
Carl Meyer f5c38382f9
gh-103963: fix 'make regen-opcode' in out-of-tree builds (#104177) 2023-05-04 17:45:56 +00:00
Erlend E. Aasland bf0b8a9f8d
gh-98822: Add missing test directories to TESTSUBDIRS (#103942) 2023-04-27 21:43:53 +02:00
Eric Snow 19e4f757de
gh-100227: Only Use deepfreeze for the Main Interpreter (gh-103794)
Deep-frozen code objects are cannot be shared (currently) by
interpreters, due to how adaptive specialization can modify the
bytecodes. We work around this by only using the deep-frozen objects in
the main interpreter. This does incur a performance penalty for
subinterpreters, which we may be able to resolve later.
2023-04-24 21:48:05 +00:00
Zachary Ware ed948e01bb
gh-103776: Remove explicit uses of $(SHELL) from Makefile (GH-103778)
This avoids conflicting with the shebang of the called scripts as well
as avoiding hard errors on platforms where the called script runs a
failing unchecked command in the usual course of checking since
`SHELL=/bin/sh -e` as of a90863c.

Fixes gh-103776.
2023-04-24 18:31:43 +00:00
Dong-hee Na 5d9762e7cf
gh-101525: Fix make test if the --enable-bolt enabled (gh-103574) 2023-04-17 23:14:18 +09:00
Erlend E. Aasland a210cac776
gh-103527: Add multibytecodec.h as make dep for _codecs_* (#103567) 2023-04-16 05:57:30 +02:00
Erlend E. Aasland 3d71b5ec5e
gh-103527: Add make deps for _codecs_* and _multibytecodec (#103528) 2023-04-14 20:47:13 +02:00
Mark Shannon 411b169281
GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython (GH-103083)
* The majority of the monitoring code is in instrumentation.c

* The new instrumentation bytecodes are in bytecodes.c

* legacy_tracing.c adapts the new API to the old sys.setrace and sys.setprofile APIs
2023-04-12 12:04:55 +01:00
Irit Katriel 33822d037a
gh-87092: move assembler related code from compile.c to assemble.c (#103277) 2023-04-11 11:15:09 +01:00
Michał Górny a90863c993
gh-100220: Fix error handling in make rules (GH-100328)
Set `SHELL = /bin/sh -e` to ensure that complex recipes fail on the first error rather than incorrectly reporting success.

Co-authored-by: Zachary Ware <zach@python.org>
2023-04-07 13:23:59 -05:00
Eric Snow 03089fdccc
gh-101659: Add _Py_AtExit() (gh-103298)
The function is like Py_AtExit() but for a single interpreter.  This is a companion to the atexit module's register() function, taking a C callback instead of a Python one.

We also update the _xxinterpchannels module to use _Py_AtExit(), which is the motivating case.  (This is inspired by pain points felt while working on gh-101660.)
2023-04-05 18:42:02 -06:00
Dong-hee Na a62ff97075
gh-101525: Disable peephole optimization process of BOLT (gh-103187)
Co-authored-by: Dong-hee Na <donghee.na@linecorp.com>
2023-04-05 09:10:45 +09:00
Irit Katriel 80163e17d3
gh-87092: move CFG related code from compile.c to flowgraph.c (#103021) 2023-03-31 18:17:59 +01:00
Eric Snow ad77d16a62
gh-102304: Move _Py_RefTotal to _PyRuntimeState (gh-102543)
The essentially eliminates the global variable, with the associated benefits. This is also a precursor to isolating this bit of state to PyInterpreterState.

Folks that currently read _Py_RefTotal directly would have to start using _Py_GetGlobalRefTotal() instead.

https://github.com/python/cpython/issues/102304
2023-03-20 10:03:04 -06:00
Guido van Rossum 70185de1ab
gh-102654: Insert #line directives in generated_cases.c.h (#102669)
This behavior is optional, because in some extreme cases it
may just make debugging harder. The tool defaults it to off,
but it is on in Makefile.pre.in.

Also note that this makes diffs to generated_cases.c.h noisier,
since whenever you insert or delete a line in bytecodes.c,
all subsequent #line directives will change.
2023-03-15 08:37:36 -07:00
Eric Snow 1ff81c0cb6
gh-81057: Add a CI Check for New Unsupported C Global Variables (gh-102506)
This will keep us from adding new unsupported (i.e. non-const) C global variables, which would break interpreter isolation.

FYI, historically it is very uncommon for new global variables to get added. Furthermore, it is rare for new code to break the c-analyzer. So the check should almost always pass unnoticed.

Note that I've removed test_check_c_globals. A test wasn't a great fit conceptually and was super slow on debug builds. A CI check is a better fit.

This also resolves gh-100237.

https://github.com/python/cpython/issues/81057
2023-03-14 10:05:54 -06:00
Jacob Bower 8de59c1bb9
gh-102021 : Allow multiple input files for interpreter loop generator (#102022)
The input files no longer use `-i`.
2023-03-03 20:59:21 -08:00
Jonathan Protzenko fcadc7e405
gh-99108: Import MD5 and SHA1 from HACL* (#102089)
Replaces our fallback non-OpenSSL MD5 and SHA1 implementations with those from HACL* as we've already done with SHA2.
2023-02-22 13:18:43 -08:00
Jason R. Coombs 072935951f
gh-97930: Also include subdirectory in makefile. (#102030) 2023-02-18 21:32:50 -05:00
Gregory P. Smith 0b13575e74
gh-99108: Refactor _sha256 & _sha512 into _sha2. (#101924)
This merges their code. They're backed by the same single HACL* static library, having them be a single module simplifies maintenance.

This should unbreak the wasm enscripten builds that currently fail due to linking in --whole-archive mode and the HACL* library appearing twice.

Long unnoticed error fixed: _sha512.SHA384Type was doubly assigned and was actually SHA512Type. Nobody depends on those internal names.

Also rename LIBHACL_ make vars to LIBHACL_SHA2_ in preperation for other future HACL things.
2023-02-15 22:08:20 -08:00
Gregory P. Smith d777790bab
gh-99108: Build the hashlib HACL* code as a static library. (#101917)
This builds HACL* as a library in one place.

A followup to #101707 which broke some WASM builds. This fixes 2/4 of them, but the enscripten toolchain in the others don't deduplicate linker arguments and error out. A follow-on PR will address those.
2023-02-14 15:57:01 -08:00
Jonathan Protzenko e5da9ab2c8
gh-99108: Import SHA2-384/512 from HACL* (#101707)
Replace the builtin hashlib implementations of SHA2-384 and SHA2-512
originally from LibTomCrypt with formally verified, side-channel resistant
code from the [HACL*](https://github.com/hacl-star/hacl-star/) project.
The builtins remain a fallback only used when OpenSSL does not provide them.
2023-02-14 01:25:16 -08:00
Guido van Rossum 65b7b6bd23
gh-98831: Use opcode metadata for stack_effect() (#101704)
* Write output and metadata in a single run
  This halves the time to run the cases generator
  (most of the time goes into parsing the input).
* Declare or define opcode metadata based on NEED_OPCODE_TABLES
* Use generated metadata for stack_effect()
* compile.o depends on opcode_metadata.h
* Return -1 from _PyOpcode_num_popped/pushed for unknown opcode
2023-02-08 16:23:19 -08:00
Michał Górny 2a8bf25804
gh-100221: Fix creating dirs in `make sharedinstall` (GH-100329)
Fix creating install directories in `make sharedinstall` if they exist already outside `DESTDIR`.  The previous make rules assumed that the directories would be created via a dependency on a rule for `$(DESTSHARED)` that did not fire if the directory did exist outside `$(DESTDIR)`.

While technically `$(DESTDIR)` could be prepended to the rule name, moving the rules for creating directories straight into the `sharedinstall` rule seems to fit the common practices better. Since the rule explicitly checks whether the individual directories exist anyway, there seems to be no reason to rely on make determining that implicitly as well.
2023-02-08 08:50:43 -08:00
Jonathan Protzenko 1fcc0efdaa
gh-99108: Replace SHA2-224 & 256 with verified code from HACL* (#99109)
replacing hashlib primitives (for the non-OpenSSL case) with verified implementations from HACL*. This is the first PR in the series, and focuses specifically on SHA2-256 and SHA2-224.

This PR imports Hacl_Streaming_SHA2 into the Python tree. This is the HACL* implementation of SHA2, which combines a core implementation of SHA2 along with a layer of buffer management that allows updating the digest with any number of bytes. This supersedes the previous implementation in the tree.

@franziskuskiefer was kind enough to benchmark the changes: in addition to being verified (thus providing significant safety and security improvements), this implementation also provides a sizeable performance boost!

```
---------------------------------------------------------------
Benchmark                     Time             CPU   Iterations
---------------------------------------------------------------
Sha2_256_Streaming            3163 ns      3160 ns       219353     // this PR
LibTomCrypt_Sha2_256          5057 ns      5056 ns       136234     // library used by Python currently
``` 

The changes in this PR are as follows:
- import the subset of HACL* that covers SHA2-256/224 into `Modules/_hacl`
- rewire sha256module.c to use the HACL* implementation

Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-02-06 18:11:01 -08:00
Dong-hee Na 144aaa74bb
gh-101282: Update BOLT --split-functions flag not to use deprecated u… (gh-101557)
gh-101282: Update BOLT --split-functions flag not to use deprecated usage
2023-02-04 16:55:31 +09:00
Gregory Szorc 79af40a403
gh-101047: Remove vestigial references to macOS libtool options (gh-101048)
LIBTOOL_CRUFT and OTHER_LIBTOOL_OPT variables have been unused since commit
19199830f7 in 2011.
2023-01-22 17:26:42 -05:00
Guido van Rossum 1f0d0a432c
GH-98831: Move assorted macros from ceval.h to a new header (#101116) 2023-01-18 10:41:07 -08:00
Carl Meyer 0cd597fef1
gh-100764: add `pycore_frame.h` to PYTHON_HEADERS and Windows build files(#100765) 2023-01-09 14:35:56 +05:30
Guido van Rossum 14b7f00fdf
GH-98831: Update generate_cases.py: register inst, opcode_metadata.h (#100735)
(These aren't used yet, but may be coming soon,
and it's easier to keep this tool the same between branches.)

Added a sanity check for all this to compile.c.

Co-authored-by: Irit Katriel <iritkatriel@yahoo.com>
2023-01-05 13:01:07 -08:00
Mark Shannon 28187141cc
GH-99005: Add `CALL_INTRINSIC_1` instruction (GH-100771)
* Remove PRINT_EXPR instruction

* Remove STOPITERATION_ERROR instruction

* Remove IMPORT_STAR instruction
2023-01-05 16:05:51 +00:00