Commit Graph

826 Commits

Author SHA1 Message Date
Michael Droettboom 50369e6c34
gh-116996: Add pystats about _Py_uop_analyse_and_optimize (GH-116997) 2024-03-22 01:27:46 +08:00
Michael Droettboom 1a33513f99
gh-116879: Add new optimizer pystats to tables (GH-116880) 2024-03-16 23:10:43 +08:00
Nikita Sobolev 4704e55a71
gh-116576: Fix `Tools/scripts/sortperf.py` sorting the same list (#116577) 2024-03-11 09:38:04 +03:00
Michael Droettboom b05afdd5ec
gh-115168: Add pystats counter for invalidated executors (GH-115169) 2024-02-26 17:51:47 +00:00
Guido van Rossum 142502ea8d
Tier 2 cleanups and tweaks (#115534)
* Rename `_testinternalcapi.get_{uop,counter}_optimizer` to `new_*_optimizer`
* Use `_PyUOpName()` instead of` _PyOpcode_uop_name[]`
* Add `target` to executor iterator items -- `list(ex)` now returns `(opcode, oparg, target, operand)` quadruples
* Add executor methods `get_opcode()` and `get_oparg()` to get `vmdata.opcode`, `vmdata.oparg`
* Define a helper for printing uops, and unify various places where they are printed
* Add a hack to summarize_stats.py to fix legacy uop names (e.g. `POP_TOP` -> `_POP_TOP`)
* Define helpers in `test_opt.py` for accessing the set or list of opnames of an executor
2024-02-20 20:24:35 +00:00
Nikita Sobolev f9154f8f23
gh-108303: Move `Lib/test/sortperf.py` to `Tools/scripts` (#114687) 2024-02-18 10:27:14 +03:00
Michael Droettboom fbb0169731
gh-115362: Add documentation to pystats output (#115365) 2024-02-16 17:06:07 +00:00
Mark Shannon 8144661017
GH-113710: Fix updating of dict version tag and add watched dict stats (GH-115221) 2024-02-12 16:07:38 +00:00
Michael Droettboom ea3cd0498c
gh-114312: Collect stats for unlikely events (GH-114493) 2024-01-25 11:10:51 +00:00
Michael Droettboom e45bae7a45
GH-114448: Don't sort summarize_stats.py histograms by amount of change (GH-114449) 2024-01-22 11:45:15 -08:00
Serhiy Storchaka b3d2427f22
gh-58032: Do not use argparse.FileType in module CLIs and scripts (GH-113649)
Open and close files manually. It prevents from leaking files,
preliminary creation of output files, and accidental closing of stdin
and stdout.
2024-01-10 15:07:19 +02:00
Guido van Rossum 7316dfb0eb
gh-112320: Implement on-trace confidence tracking for branches (#112321)
We track the confidence as a scaled int.
2023-12-12 21:43:08 +00:00
Michael Droettboom 9495bcaf59
gh-111277: In summarize_stats.py, don't fail fast on invalid ratios (#111278) 2023-10-30 17:10:07 -07:00
Michael Droettboom 84b4533e84
gh-109329: Count tier2 opcode misses (#110561)
This keeps a separate 'miss' counter for each micro-opcode, incremented whenever a guard uop takes a deoptimization side exit.
2023-10-30 17:02:45 -07:00
Mark Shannon b0699aa544
GH-111213: Fix a few broken stats (GH-111216) 2023-10-26 11:33:12 +01:00
Michael Droettboom 81eba76450
gh-110019: Refactor summarize_stats (GH-110398) 2023-10-24 09:57:39 +01:00
Michael Droettboom 9eb2489266
gh-109329: Add stat for "trace too short" (GH-110402) 2023-10-05 16:12:06 +01:00
Michael Droettboom e561e98058
GH-109329: Add tier 2 stats (GH-109913) 2023-10-04 14:52:28 -07:00
Victor Stinner 53eb9a676f
gh-110152: regrtest handles cross compilation and HOSTRUNNER (#110156)
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
  HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
  or "make test" instead.
2023-09-30 22:37:23 +00:00
Victor Stinner fbfec5642e
gh-109566: regrtest reexecutes the process (#109909)
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
2023-09-26 20:46:52 +02:00
Victor Stinner 859618c8cd
gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
2023-09-26 17:22:50 +02:00
Michael Droettboom 19f5effc27
GH-109373: Store metadata required for pystats comparison in the JSON (GH-109374) 2023-09-15 13:10:46 -07:00
Michael Droettboom 5dcbbd8861
GH-109330: Dump and compare stats using opcode names, not numbers (GH-109335) 2023-09-12 14:12:57 -07:00
Michael Droettboom 52beebc856
gh-109136: Fix summarize_stats.py tool (#109137) 2023-09-08 18:23:58 +02:00
Irit Katriel 665a4391e1
gh-105481: generate op IDs from bytecode.c instead of hard coding them in opcode.py (#107971) 2023-08-16 22:25:18 +00:00
Mark Shannon 2ba7c7f7b1
Add some GC stats to Py_STATS (GH-107581) 2023-08-04 10:34:23 +01:00
Victor Stinner ae00b810d1
gh-104780: Remove 2to3 program and lib2to3 module (#104781)
* Remove the Tools/scripts/2to3 script.
* Remove the Lib/test/test_lib2to3/ directory.
* Doc/tools/extensions/pyspecific.py: remove the "2to3fixer" object
  type.
* Makefile and PC/layout/main.py no longer compile lib2to3 grammar
  files.
* Update Makefile for 2to3 removal.
2023-05-23 19:40:02 +02:00
Carl Meyer 77262458fe
gh-87729: improve hit rate of LOAD_SUPER_ATTR specialization (#104270) 2023-05-11 08:08:13 -06:00
penguin_wwy 373bca0cc5
GH-102181: Improve specialization stats for SEND (GH-102182) 2023-05-10 22:40:59 +00:00
Missoupro 01a49d1745
gh-102110: Add all tools description missed (GH-102625)
Added missing descriptions of some tools.

Automerge-Triggered-By: GH:hugovk
2023-03-30 13:49:07 -07:00
Brandt Bucher 0444ae2487
GH-100982: Break up COMPARE_AND_BRANCH (GH-102801) 2023-03-23 15:25:09 -07:00
Mark Shannon 7b14c2ef19
GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983) 2023-01-16 12:35:21 +00:00
Mark Shannon c00eb1eae6
Don't double count misses. (GH-100984) 2023-01-13 15:57:39 +00:00
penguin_wwy a02161286a
GH-99770: Make the correct call specialization fail kind show up in the stats (GH-99771) 2022-12-22 12:44:59 +00:00
Mark Shannon 5693f45b19
Assorted minor fixes for specialization stats. (GH-100219) 2022-12-14 15:50:02 +00:00
Michael Droettboom 1583c6e326
GH-100143: Improve collecting pystats for parts of runs (GH-100144)
* pystats off by default

* Add -Xpystats flag

* Always dump pystats, even if turned off
2022-12-12 14:50:43 +00:00
Michael Droettboom 9dc787ea96
GH-100026: Include the number of raw input files in summarize_stats.py (GH-100027) 2022-12-06 11:14:47 +00:00
Michael Droettboom bf94c653f4
GH-99905: Fix output of misses in summarize_stats.py execution counts (GH-99906)
This was an indentation error introduced in 2844aa6a
2022-11-30 19:17:08 +00:00
Michael Droettboom 2844aa6a8e
Support comparing two sets of pystats (GH-98816)
This adds support for comparing pystats collected from two different builds.

- The `--json-output` can be used to load in a set of raw stats and output a
  JSON file.
- Two of these JSON files can be provided on the next run, and then comparative
  results between the two are output.
2022-11-04 10:15:54 +00:00
Neil Schemenauer de6981680b
gh-90716: add _pylong.py module (#96673)
Add Python implementations of certain longobject.c functions. These use
asymptotically faster algorithms that can be used for operations on
integers with many digits. In those cases, the performance overhead of
the Python implementation is not significant since the asymptotic
behavior is what dominates runtime. Functions provided by this module
should be considered private and not part of any public API.

Co-author: Tim Peters <tim.peters@gmail.com>
Co-author: Mark Dickinson <dickinsm@gmail.com>
Co-author: Bjorn Martinsson
2022-10-25 22:00:50 -07:00
Victor Stinner 1863302d61
gh-97669: Create Tools/build/ directory (#97963)
Create Tools/build/ directory. Move the following scripts from
Tools/scripts/ to Tools/build/:

* check_extension_modules.py
* deepfreeze.py
* freeze_modules.py
* generate_global_objects.py
* generate_levenshtein_examples.py
* generate_opcode_h.py
* generate_re_casefix.py
* generate_sre_constants.py
* generate_stdlib_module_names.py
* generate_token.py
* parse_html5_entities.py
* smelly.py
* stable_abi.py
* umarshal.py
* update_file.py
* verify_ensurepip_wheels.py

Update references to these scripts.
2022-10-17 12:01:00 +02:00
Victor Stinner d09d2c7c91
gh-97669: Remove Tools/scripts/startuptime.py (#98214)
The "pyperf command" tool be used instead. Example:

    $ python3 -m pyperf command -- python3 -c pass
    .....................
    command: Mean +- std dev: 17.8 ms +- 0.4 ms

pyperf also computes the standard deviation which gives an idea of
the benchmark looks reliable or not.
2022-10-12 17:53:46 +02:00
Victor Stinner 0895c2a066
gh-97669: Create Tools/patchcheck/ directory (#98186)
Move patchcheck.py, reindent.py and untabify.py scripts to a new
Tools/patchcheck/ directory.
2022-10-12 10:09:21 +02:00
Victor Stinner 454a6d61bc
gh-97669: Remove abitype.py and pep384_macrocheck.py (#98165)
Remove abitype.py and pep384_macrocheck.py scripts of Tools/scripts/.
2022-10-11 13:18:53 +02:00
Victor Stinner e0ae9ddffe
gh-97669: Remove outdated example scripts (#97675) (#98167)
Remove outdated example scripts of the Tools/scripts/ directory:

* gprof2html.py
* md5sum.py
* nm2def.py
* pathfix.py
* win_add2path.py

Remove test_gprof2html, test_md5sum and test_pathfix of test_tools.
2022-10-11 10:07:57 +02:00
Ken Jin b399115ef1
gh-95756: Lazily created cached co_* attrs (GH-97791) 2022-10-11 11:26:08 +08:00
Joannah Nanjekye 4ed00be98f
gh-68686: Retire eptag ptag scripts (#98064)
* Retire eptag ptag scripts

* 📜🤖 Added by blurb_it.

* fix news entry error

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-10-08 07:57:47 -07:00
Victor Stinner 002252c4ad
gh-97669: Move difflib examples to Doc/includes/ (#97964)
Remove diff.py and ndiff.py scripts of Tools/scripts/: move them to
Doc/includes/.

* diff.py and ndiff.py files are no longer executable. Remove also
  their shebang ("#!/usr/bin/env python3").
* Remove the -profile command from ndiff.py to simply the code.
* Remove ndiff.py copyright and history command. The Python
  documentation examples are distributed under the "Zero Clause BSD
  License".
2022-10-07 10:57:48 -07:00
Łukasz Langa bbc7cd649a
gh-97008: Add a Python implementation of AttributeError and NameError suggestions (#97022)
Relevant tests moved from test_exceptions to test_traceback to be able to
compare both implementations.

Co-authored-by: Carl Friedrich Bolz-Tereick <cfbolz@gmx.de>
2022-10-04 15:31:16 -07:00
Victor Stinner 116fa62c6e
gh-97670: Remove sys.getdxp() and analyze_dxp.py script (#97671)
Remove the sys.getdxp() function and the Tools/scripts/analyze_dxp.py
script. DXP stands for "dynamic execution pairs". They were related
to DYNAMIC_EXECUTION_PROFILE and DXPAIRS macros which have been
removed in Python 3.11. Python can now be built with "./configure
--enable-pystats" to gather statistics on Python opcodes.
2022-10-04 15:28:57 +02:00