Commit Graph

9 Commits

Author SHA1 Message Date
Miss Islington (bot) 0e4d526de4
bpo-41811: create SortKey members using first given value (GH-22316) (GH-22325)
(cherry picked from commit ae0d2a33ec)

Co-authored-by: Ethan Furman <ethan@stoneleaf.us>
2020-09-19 12:56:13 -07:00
Victor Stinner 57572b103e
bpo-40443: Remove unused imports in tests (GH-19805) 2020-04-30 01:48:37 +02:00
Daniel Olshansky 01602ae403 bpo-37958: Adding get_profile_dict to pstats (GH-15495)
pstats is really useful or profiling and printing the output of the execution of some block of code, but I've found on multiple occasions when I'd like to access this output directly in an easily usable dictionary on which I can further analyze or manipulate.

The proposal is to add a function called get_profile_dict inside of pstats that'll automatically return this data the data in an easily accessible dict.

The output of the following script:

```
import cProfile, pstats
import pprint
from pstats import func_std_string, f8

def fib(n):
    if n == 0:
        return 0
    if n == 1:
        return 1
    return fib(n-1) + fib(n-2)

pr = cProfile.Profile()
pr.enable()
fib(5)
pr.create_stats()

ps = pstats.Stats(pr).sort_stats('tottime', 'cumtime')

def get_profile_dict(self, keys_filter=None):
    """
        Returns a dict where the key is a function name and the value is a dict
        with the following keys:
            - ncalls
            - tottime
            - percall_tottime
            - cumtime
            - percall_cumtime
            - file_name
            - line_number

        keys_filter can be optionally set to limit the key-value pairs in the
        retrieved dict.
    """
    pstats_dict = {}
    func_list = self.fcn_list[:] if self.fcn_list else list(self.stats.keys())

    if not func_list:
        return pstats_dict

    pstats_dict["total_tt"] = float(f8(self.total_tt))
    for func in func_list:
        cc, nc, tt, ct, callers = self.stats[func]
        file, line, func_name = func
        ncalls = str(nc) if nc == cc else (str(nc) + '/' + str(cc))
        tottime = float(f8(tt))
        percall_tottime = -1 if nc == 0 else float(f8(tt/nc))
        cumtime = float(f8(ct))
        percall_cumtime = -1 if cc == 0 else float(f8(ct/cc))
        func_dict = {
            "ncalls": ncalls,
            "tottime": tottime, # time spent in this function alone
            "percall_tottime": percall_tottime,
            "cumtime": cumtime, # time spent in the function plus all functions that this function called,
            "percall_cumtime": percall_cumtime,
            "file_name": file,
            "line_number": line
        }
        func_dict_filtered = func_dict if not keys_filter else { key: func_dict[key] for key in keys_filter }
        pstats_dict[func_name] = func_dict_filtered

    return pstats_dict

pp = pprint.PrettyPrinter(depth=6)
pp.pprint(get_profile_dict(ps))
```

will produce:

```
{"<method 'disable' of '_lsprof.Profiler' objects>": {'cumtime': 0.0,
                                                      'file_name': '~',
                                                      'line_number': 0,
                                                      'ncalls': '1',
                                                      'percall_cumtime': 0.0,
                                                      'percall_tottime': 0.0,
                                                      'tottime': 0.0},
 'create_stats': {'cumtime': 0.0,
                  'file_name': '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/cProfile.py',
                  'line_number': 50,
                  'ncalls': '1',
                  'percall_cumtime': 0.0,
                  'percall_tottime': 0.0,
                  'tottime': 0.0},
 'fib': {'cumtime': 0.0,
         'file_name': 'get_profile_dict.py',
         'line_number': 5,
         'ncalls': '15/1',
         'percall_cumtime': 0.0,
         'percall_tottime': 0.0,
         'tottime': 0.0},
 'total_tt': 0.0}
 ```

 As an example, this can be used to generate a stacked column chart using various visualization tools which will assist in easily identifying program bottlenecks.



https://bugs.python.org/issue37958



Automerge-Triggered-By: @gpshead
2020-01-15 14:51:54 -08:00
mwidjaja 863b1e4d0e bpo-29237: Create enum for pstats sorting options (GH-5103) 2018-01-25 20:49:56 -08:00
Zachary Ware 38c707e7e0 Issue #21741: Update 147 test modules to use test discovery.
I have compared output between pre- and post-patch runs of these tests
to make sure there's nothing missing and nothing broken, on both
Windows and Linux.  The only differences I found were actually tests
that were previously *not* run.
2015-04-13 15:00:43 -05:00
Georg Brandl 83938437cb #10166: rewrite self-recursion to iteration in pstats.Stats.add(). Also add a unittest and a stats test file. 2010-10-22 06:28:01 +00:00
Georg Brandl 2d3c4e79a1 #7372: fix regression in pstats: a previous fix to handle cProfile data in add_callers broke handling of profile data. 2010-08-02 17:24:49 +00:00
Benjamin Peterson ee8712cda4 #2621 rename test.test_support to test.support 2008-05-20 21:35:26 +00:00
Christian Heimes e1c981161c Merged revisions 60143-60149 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r60143 | georg.brandl | 2008-01-20 15:50:05 +0100 (Sun, 20 Jan 2008) | 3 lines

  Switch mmap from old Py_FindMethod to new PyObject_GenericGetAttr attribute access.
  Fixes #1087735.
........
  r60145 | georg.brandl | 2008-01-20 20:40:58 +0100 (Sun, 20 Jan 2008) | 2 lines

  Add blurb about executable scripts on Windows. #760657.
........
  r60146 | georg.brandl | 2008-01-20 20:48:40 +0100 (Sun, 20 Jan 2008) | 2 lines

  #1219903: fix tp_richcompare docs.
........
  r60147 | georg.brandl | 2008-01-20 22:10:08 +0100 (Sun, 20 Jan 2008) | 2 lines

  Fix markup.
........
  r60148 | gregory.p.smith | 2008-01-21 08:11:11 +0100 (Mon, 21 Jan 2008) | 14 lines

  Provide a sanity check during PyThreadState_DeleteCurrent() and
  PyThreadState_Delete() to avoid an infinite loop when the tstate list
  is messed up and has somehow becomes circular and does not contain the
  current thread.

  I don't know how this happens but it does, *very* rarely.  On more than
  one hardware platform.  I have not been able to reproduce it manually.

  Attaching to a process where its happening: it has always been in an
  infinite loop over a single element tstate list that is not the tstate
  we're looking to delete.  It has been in t_bootstrap()'s call to
  PyThreadState_DeleteCurrent() as a pthread is exiting.
........
  r60149 | georg.brandl | 2008-01-21 11:24:59 +0100 (Mon, 21 Jan 2008) | 2 lines

  #1269: fix a bug in pstats.add_callers() and add a unit test file for pstats.
........
2008-01-21 11:20:28 +00:00