pstats is really useful or profiling and printing the output of the execution of some block of code, but I've found on multiple occasions when I'd like to access this output directly in an easily usable dictionary on which I can further analyze or manipulate.
The proposal is to add a function called get_profile_dict inside of pstats that'll automatically return this data the data in an easily accessible dict.
The output of the following script:
```
import cProfile, pstats
import pprint
from pstats import func_std_string, f8
def fib(n):
if n == 0:
return 0
if n == 1:
return 1
return fib(n-1) + fib(n-2)
pr = cProfile.Profile()
pr.enable()
fib(5)
pr.create_stats()
ps = pstats.Stats(pr).sort_stats('tottime', 'cumtime')
def get_profile_dict(self, keys_filter=None):
"""
Returns a dict where the key is a function name and the value is a dict
with the following keys:
- ncalls
- tottime
- percall_tottime
- cumtime
- percall_cumtime
- file_name
- line_number
keys_filter can be optionally set to limit the key-value pairs in the
retrieved dict.
"""
pstats_dict = {}
func_list = self.fcn_list[:] if self.fcn_list else list(self.stats.keys())
if not func_list:
return pstats_dict
pstats_dict["total_tt"] = float(f8(self.total_tt))
for func in func_list:
cc, nc, tt, ct, callers = self.stats[func]
file, line, func_name = func
ncalls = str(nc) if nc == cc else (str(nc) + '/' + str(cc))
tottime = float(f8(tt))
percall_tottime = -1 if nc == 0 else float(f8(tt/nc))
cumtime = float(f8(ct))
percall_cumtime = -1 if cc == 0 else float(f8(ct/cc))
func_dict = {
"ncalls": ncalls,
"tottime": tottime, # time spent in this function alone
"percall_tottime": percall_tottime,
"cumtime": cumtime, # time spent in the function plus all functions that this function called,
"percall_cumtime": percall_cumtime,
"file_name": file,
"line_number": line
}
func_dict_filtered = func_dict if not keys_filter else { key: func_dict[key] for key in keys_filter }
pstats_dict[func_name] = func_dict_filtered
return pstats_dict
pp = pprint.PrettyPrinter(depth=6)
pp.pprint(get_profile_dict(ps))
```
will produce:
```
{"<method 'disable' of '_lsprof.Profiler' objects>": {'cumtime': 0.0,
'file_name': '~',
'line_number': 0,
'ncalls': '1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'create_stats': {'cumtime': 0.0,
'file_name': '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/cProfile.py',
'line_number': 50,
'ncalls': '1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'fib': {'cumtime': 0.0,
'file_name': 'get_profile_dict.py',
'line_number': 5,
'ncalls': '15/1',
'percall_cumtime': 0.0,
'percall_tottime': 0.0,
'tottime': 0.0},
'total_tt': 0.0}
```
As an example, this can be used to generate a stacked column chart using various visualization tools which will assist in easily identifying program bottlenecks.
https://bugs.python.org/issue37958
Automerge-Triggered-By: @gpshead
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r86580 | senthil.kumaran | 2010-11-21 01:02:50 +0800 (Sun, 21 Nov 2010) | 3 lines
Fix issue10377 - Output from pstats - it is just secs (i.e, wallclock time) and not CPU time.
........
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r83531 | georg.brandl | 2010-08-02 19:24:49 +0200 (Mo, 02 Aug 2010) | 1 line
#7372: fix regression in pstats: a previous fix to handle cProfile data in add_callers broke handling of profile data.
........
svn+ssh://svn.python.org/python/branches/py3k
........
r83385 | georg.brandl | 2010-08-01 08:42:45 +0200 (So, 01 Aug 2010) | 1 line
#8773: mailbox.py does not need to be executable.
........
r83386 | georg.brandl | 2010-08-01 08:44:46 +0200 (So, 01 Aug 2010) | 1 line
#8768: name test method properly so that it gets executed.
........
r83387 | georg.brandl | 2010-08-01 08:53:28 +0200 (So, 01 Aug 2010) | 1 line
#8735: better explain semantics of *values* argument for parse().
........
r83388 | georg.brandl | 2010-08-01 09:48:43 +0200 (So, 01 Aug 2010) | 1 line
#7395: fix traceback in do_add() when no stats are loaded. Apply same fix for do_sort() and do_reverse().
........
r83389 | georg.brandl | 2010-08-01 09:57:47 +0200 (So, 01 Aug 2010) | 1 line
Small improvements to pstats browser: do not crash on reading invalid file, and actually do a reload when executing "read" as intended.
........
r83391 | georg.brandl | 2010-08-01 10:10:08 +0200 (So, 01 Aug 2010) | 1 line
Add another news entry.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r79603 | andrew.kuchling | 2010-04-02 12:59:16 -0400 (Fri, 02 Apr 2010) | 1 line
#4440: modernize a use of filter(), making it compatible with 3.x
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r60143 | georg.brandl | 2008-01-20 15:50:05 +0100 (Sun, 20 Jan 2008) | 3 lines
Switch mmap from old Py_FindMethod to new PyObject_GenericGetAttr attribute access.
Fixes#1087735.
........
r60145 | georg.brandl | 2008-01-20 20:40:58 +0100 (Sun, 20 Jan 2008) | 2 lines
Add blurb about executable scripts on Windows. #760657.
........
r60146 | georg.brandl | 2008-01-20 20:48:40 +0100 (Sun, 20 Jan 2008) | 2 lines
#1219903: fix tp_richcompare docs.
........
r60147 | georg.brandl | 2008-01-20 22:10:08 +0100 (Sun, 20 Jan 2008) | 2 lines
Fix markup.
........
r60148 | gregory.p.smith | 2008-01-21 08:11:11 +0100 (Mon, 21 Jan 2008) | 14 lines
Provide a sanity check during PyThreadState_DeleteCurrent() and
PyThreadState_Delete() to avoid an infinite loop when the tstate list
is messed up and has somehow becomes circular and does not contain the
current thread.
I don't know how this happens but it does, *very* rarely. On more than
one hardware platform. I have not been able to reproduce it manually.
Attaching to a process where its happening: it has always been in an
infinite loop over a single element tstate list that is not the tstate
we're looking to delete. It has been in t_bootstrap()'s call to
PyThreadState_DeleteCurrent() as a pthread is exiting.
........
r60149 | georg.brandl | 2008-01-21 11:24:59 +0100 (Mon, 21 Jan 2008) | 2 lines
#1269: fix a bug in pstats.add_callers() and add a unit test file for pstats.
........
and .keys(), .items(), .values() return dict views.
The dict views aren't fully functional yet; in particular, they can't
be compared to sets yet. but they are useful as "iterator wells".
There are still 27 failing unit tests; I expect that many of these
have fairly trivial fixes, but there are so many, I could use help.
There's one major and one minor category still unfixed:
doctests are the major category (and I hope to be able to augment the
refactoring tool to refactor bona fide doctests soon);
other code generating print statements in strings is the minor category.
(Oh, and I don't know if the compiler package works.)
Miss Islington (bot)
Daniel Olshansky
Xtreak
penguindustin
Serhiy Storchaka
mwidjaja
Jon Dufresne
Matthias Bussonnier
Stefan Krah
Brett Cannon
Giampaolo Rodola'
Andrew Svetlov
Antoine Pitrou
Florent Xicluna
Éric Araujo
Benjamin Peterson
Ezio Melotti
Senthil Kumaran
Georg Brandl
Raymond Hettinger
Andrew M. Kuchling
Christian Heimes
Guido van Rossum
Collin Winter