Commit Graph

87 Commits

Author SHA1 Message Date
Ethan Furman ae0d2a33ec
bpo-41811: create SortKey members using first given value (GH-22316) 2020-09-19 11:12:57 -07:00
Daniel Olshansky 01602ae403 bpo-37958: Adding get_profile_dict to pstats (GH-15495)
pstats is really useful or profiling and printing the output of the execution of some block of code, but I've found on multiple occasions when I'd like to access this output directly in an easily usable dictionary on which I can further analyze or manipulate.

The proposal is to add a function called get_profile_dict inside of pstats that'll automatically return this data the data in an easily accessible dict.

The output of the following script:

```
import cProfile, pstats
import pprint
from pstats import func_std_string, f8

def fib(n):
    if n == 0:
        return 0
    if n == 1:
        return 1
    return fib(n-1) + fib(n-2)

pr = cProfile.Profile()
pr.enable()
fib(5)
pr.create_stats()

ps = pstats.Stats(pr).sort_stats('tottime', 'cumtime')

def get_profile_dict(self, keys_filter=None):
    """
        Returns a dict where the key is a function name and the value is a dict
        with the following keys:
            - ncalls
            - tottime
            - percall_tottime
            - cumtime
            - percall_cumtime
            - file_name
            - line_number

        keys_filter can be optionally set to limit the key-value pairs in the
        retrieved dict.
    """
    pstats_dict = {}
    func_list = self.fcn_list[:] if self.fcn_list else list(self.stats.keys())

    if not func_list:
        return pstats_dict

    pstats_dict["total_tt"] = float(f8(self.total_tt))
    for func in func_list:
        cc, nc, tt, ct, callers = self.stats[func]
        file, line, func_name = func
        ncalls = str(nc) if nc == cc else (str(nc) + '/' + str(cc))
        tottime = float(f8(tt))
        percall_tottime = -1 if nc == 0 else float(f8(tt/nc))
        cumtime = float(f8(ct))
        percall_cumtime = -1 if cc == 0 else float(f8(ct/cc))
        func_dict = {
            "ncalls": ncalls,
            "tottime": tottime, # time spent in this function alone
            "percall_tottime": percall_tottime,
            "cumtime": cumtime, # time spent in the function plus all functions that this function called,
            "percall_cumtime": percall_cumtime,
            "file_name": file,
            "line_number": line
        }
        func_dict_filtered = func_dict if not keys_filter else { key: func_dict[key] for key in keys_filter }
        pstats_dict[func_name] = func_dict_filtered

    return pstats_dict

pp = pprint.PrettyPrinter(depth=6)
pp.pprint(get_profile_dict(ps))
```

will produce:

```
{"<method 'disable' of '_lsprof.Profiler' objects>": {'cumtime': 0.0,
                                                      'file_name': '~',
                                                      'line_number': 0,
                                                      'ncalls': '1',
                                                      'percall_cumtime': 0.0,
                                                      'percall_tottime': 0.0,
                                                      'tottime': 0.0},
 'create_stats': {'cumtime': 0.0,
                  'file_name': '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/cProfile.py',
                  'line_number': 50,
                  'ncalls': '1',
                  'percall_cumtime': 0.0,
                  'percall_tottime': 0.0,
                  'tottime': 0.0},
 'fib': {'cumtime': 0.0,
         'file_name': 'get_profile_dict.py',
         'line_number': 5,
         'ncalls': '15/1',
         'percall_cumtime': 0.0,
         'percall_tottime': 0.0,
         'tottime': 0.0},
 'total_tt': 0.0}
 ```

 As an example, this can be used to generate a stacked column chart using various visualization tools which will assist in easily identifying program bottlenecks.



https://bugs.python.org/issue37958



Automerge-Triggered-By: @gpshead
2020-01-15 14:51:54 -08:00
Xtreak 0d70227e41 Fix typos in docs and docstrings (GH-13745) 2019-06-03 01:12:33 +02:00
penguindustin 9646630895 bpo-36766: Typos in docs and code comments (GH-13116) 2019-05-06 14:57:17 -04:00
Serhiy Storchaka 3f2e6f15d6
Revert unneccessary changes made in bpo-30296 and apply other improvements. (GH-2624) 2018-02-26 16:50:11 +02:00
mwidjaja 863b1e4d0e bpo-29237: Create enum for pstats sorting options (GH-5103) 2018-01-25 20:49:56 -08:00
Jon Dufresne 3972628de3 bpo-30296 Remove unnecessary tuples, lists, sets, and dicts (#1489)
* Replaced list(<generator expression>) with list comprehension
* Replaced dict(<generator expression>) with dict comprehension
* Replaced set(<list literal>) with set literal
* Replaced builtin func(<list comprehension>) with func(<generator
  expression>) when supported (e.g. any(), all(), tuple(), min(), &
  max())
2017-05-18 07:35:54 -07:00
Serhiy Storchaka 55fe1ae970 bpo-30022: Get rid of using EnvironmentError and IOError (except test… (#1051) 2017-04-16 10:46:38 +03:00
Matthias Bussonnier 8fb1f6e039 bpo-29554: Improve docs for pstat module and profile. (#88)
Clarify that methods take a string which is interpreted as a regex,
not a regex object.

Also clarify what the old `-1`, `0`, `1` and `2` options were.
2017-02-21 08:30:00 +03:00
Stefan Krah e12a68be35 Issue #27241: Catch exception when running pstats as main. 2016-08-02 22:30:24 +02:00
Brett Cannon cd171c8e92 Issue #18200: Back out usage of ModuleNotFoundError (8d28d44f3a9a) 2013-07-04 17:43:24 -04:00
Brett Cannon 0a140668fa Issue #18200: Update the stdlib (except tests) to use
ModuleNotFoundError.
2013-06-13 20:57:26 -04:00
Giampaolo Rodola' 2f50aaf2ff modernize some modules' code by using with statement around open() 2013-02-12 02:04:27 +01:00
Andrew Svetlov f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
Andrew Svetlov 3e5dae0ee2 Merge issue #14900: Add aliases for sorting params for pstat to follow column names from pstat output.
Patch by Arne Babenhauserheide.
2012-10-07 19:19:17 +03:00
Andrew Svetlov c3e5b10ae7 Issue #14900: Add aliases for sorting params for pstat to follow column names from pstat output.
Patch by Arne Babenhauserheide.
2012-10-07 19:18:39 +03:00
Andrew Svetlov a514ea32b2 Merge issue #14900: Distuguish call count and primitive call count in pstat output.
Patch by Arne Babenhauserheide.
2012-10-07 18:59:11 +03:00
Andrew Svetlov 2ef4584e6d Issue #14900: Distuguish call count and primitive call count in pstat output.
Patch by Arne Babenhauserheide.
2012-10-07 18:58:42 +03:00
Antoine Pitrou 9d8c186650 Issue #13839: When invoked on the command-line, the pstats module now accepts several filenames of profile stat files and merges them all.
Patch by Matt Joiner.
2012-03-14 17:47:11 +01:00
Florent Xicluna 54540ec917 Remove redundant imports. 2011-11-04 08:29:17 +01:00
Éric Araujo fc662ddda2 Merge profile docs followup (#12417) from 3.1 2011-07-28 23:00:14 +02:00
Éric Araujo ee19c772cb Remove mentions of previous license in profile module docs (#12417 followup).
Also remove an extra docstring.
2011-07-28 22:56:24 +02:00
Benjamin Peterson 0f93d3d6f3 strip trailing ws 2011-06-27 09:18:46 -05:00
Benjamin Peterson afa44a8096 update profile license (closes #12417) 2011-06-27 09:14:34 -05:00
Benjamin Peterson 8d77069558 update profile license (closes #12417) 2011-06-27 09:14:34 -05:00
Ezio Melotti 13925008dc #11565: Fix several typos. Patch by Piotr Kasprzyk. 2011-03-16 11:05:33 +02:00
Senthil Kumaran e062ba48b6 Merged revisions 86580 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r86580 | senthil.kumaran | 2010-11-21 01:02:50 +0800 (Sun, 21 Nov 2010) | 3 lines

  Fix issue10377 - Output from pstats - it is just secs (i.e, wallclock time) and not CPU time.
........
2010-11-20 17:08:19 +00:00
Senthil Kumaran 5e703cf32b Fix issue10377 - Output from pstats - it is just secs (i.e, wallclock time) and not CPU time. 2010-11-20 17:02:50 +00:00
Georg Brandl 9a8439d3e9 Refactor interesting use of try-finally. 2010-10-22 06:35:59 +00:00
Georg Brandl eb7e56922e Make top_level attribute a set instead of a dict with None values. 2010-10-22 06:29:21 +00:00
Georg Brandl 83938437cb #10166: rewrite self-recursion to iteration in pstats.Stats.add(). Also add a unittest and a stats test file. 2010-10-22 06:28:01 +00:00
Georg Brandl 4009c9edfc Merged revisions 82805-82806,83523-83527,83536,83538,83542,83546-83548,83550-83555,83558,83560 via svnmerge from
svn+ssh://svn.python.org/python/branches/py3k

........
  r82805 | georg.brandl | 2010-07-11 11:42:10 +0200 (So, 11 Jul 2010) | 1 line

  #7935: cross-reference to ast.literal_eval() from eval() docs.
........
  r82806 | georg.brandl | 2010-07-11 12:22:44 +0200 (So, 11 Jul 2010) | 1 line

  #9223: link to Command class reference, and move Command interface docs nearer to class docs.
........
  r83523 | georg.brandl | 2010-08-02 14:06:18 +0200 (Mo, 02 Aug 2010) | 1 line

  #9209 and #7781: fix two crashes in pstats interactive browser.
........
  r83524 | georg.brandl | 2010-08-02 14:20:23 +0200 (Mo, 02 Aug 2010) | 1 line

  #9428: fix running scripts from profile/cProfile with their own name and the right namespace.  Same fix as for trace.py in #1690103.
........
  r83525 | georg.brandl | 2010-08-02 14:36:24 +0200 (Mo, 02 Aug 2010) | 1 line

  Get rid of spurious "threading" entries in trace output.
........
  r83526 | georg.brandl | 2010-08-02 14:40:22 +0200 (Mo, 02 Aug 2010) | 1 line

  Fix softspace relic.
........
  r83527 | georg.brandl | 2010-08-02 14:48:46 +0200 (Mo, 02 Aug 2010) | 1 line

  #3821: beginnings of a trace.py unittest.
........
  r83536 | georg.brandl | 2010-08-02 19:49:25 +0200 (Mo, 02 Aug 2010) | 1 line

  #8578: mention danger of not incref'ing weak referenced object.
........
  r83538 | georg.brandl | 2010-08-02 20:10:13 +0200 (Mo, 02 Aug 2010) | 1 line

  #6928: fix class docs w.r.t. new metaclasses.
........
  r83542 | georg.brandl | 2010-08-02 20:56:54 +0200 (Mo, 02 Aug 2010) | 1 line

  Move test_SimpleHTTPServer into test_httpservers.
........
  r83546 | georg.brandl | 2010-08-02 21:16:34 +0200 (Mo, 02 Aug 2010) | 1 line

  #7973: Fix distutils options spelling.
........
  r83547 | georg.brandl | 2010-08-02 21:19:26 +0200 (Mo, 02 Aug 2010) | 1 line

  #7386: add example that shows that trailing path separators are stripped.
........
  r83548 | georg.brandl | 2010-08-02 21:23:34 +0200 (Mo, 02 Aug 2010) | 1 line

  #8172: how does one use a property?
........
  r83550 | georg.brandl | 2010-08-02 21:32:43 +0200 (Mo, 02 Aug 2010) | 1 line

  #9451: strengthen warning about __*__ special name usage.
........
  r83551 | georg.brandl | 2010-08-02 21:35:06 +0200 (Mo, 02 Aug 2010) | 1 line

  Remove XXX comment that was displayed.
........
  r83552 | georg.brandl | 2010-08-02 21:36:36 +0200 (Mo, 02 Aug 2010) | 1 line

  #9438: clarify that constant names also cannot be assigned as attributes.
........
  r83553 | georg.brandl | 2010-08-02 21:39:17 +0200 (Mo, 02 Aug 2010) | 1 line

  Remove redundant information.
........
  r83554 | georg.brandl | 2010-08-02 21:43:05 +0200 (Mo, 02 Aug 2010) | 1 line

  #7280: note about nasmw.exe.
........
  r83555 | georg.brandl | 2010-08-02 21:44:48 +0200 (Mo, 02 Aug 2010) | 1 line

  #8861: remove unused variable.
........
  r83558 | georg.brandl | 2010-08-02 22:05:19 +0200 (Mo, 02 Aug 2010) | 1 line

  #8648: document UTF-7 codec functions.
........
  r83560 | georg.brandl | 2010-08-02 22:16:18 +0200 (Mo, 02 Aug 2010) | 1 line

  #9087: update json docstrings -- unicode and long do not exist anymore.
........
2010-10-06 08:26:09 +00:00
Georg Brandl e90bce7df7 Merged revisions 83531 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r83531 | georg.brandl | 2010-08-02 19:24:49 +0200 (Mo, 02 Aug 2010) | 1 line

  #7372: fix regression in pstats: a previous fix to handle cProfile data in add_callers broke handling of profile data.
........
2010-08-02 17:36:05 +00:00
Georg Brandl 2d3c4e79a1 #7372: fix regression in pstats: a previous fix to handle cProfile data in add_callers broke handling of profile data. 2010-08-02 17:24:49 +00:00
Georg Brandl b1a97afadb #9209 and #7781: fix two crashes in pstats interactive browser. 2010-08-02 12:06:18 +00:00
Georg Brandl 44c58236d7 Merged revisions 83385-83389,83391 via svnmerge from
svn+ssh://svn.python.org/python/branches/py3k

........
  r83385 | georg.brandl | 2010-08-01 08:42:45 +0200 (So, 01 Aug 2010) | 1 line

  #8773: mailbox.py does not need to be executable.
........
  r83386 | georg.brandl | 2010-08-01 08:44:46 +0200 (So, 01 Aug 2010) | 1 line

  #8768: name test method properly so that it gets executed.
........
  r83387 | georg.brandl | 2010-08-01 08:53:28 +0200 (So, 01 Aug 2010) | 1 line

  #8735: better explain semantics of *values* argument for parse().
........
  r83388 | georg.brandl | 2010-08-01 09:48:43 +0200 (So, 01 Aug 2010) | 1 line

  #7395: fix traceback in do_add() when no stats are loaded.  Apply same fix for do_sort() and do_reverse().
........
  r83389 | georg.brandl | 2010-08-01 09:57:47 +0200 (So, 01 Aug 2010) | 1 line

  Small improvements to pstats browser: do not crash on reading invalid file, and actually do a reload when executing "read" as intended.
........
  r83391 | georg.brandl | 2010-08-01 10:10:08 +0200 (So, 01 Aug 2010) | 1 line

  Add another news entry.
........
2010-08-01 19:04:55 +00:00
Georg Brandl f02e7367eb Small improvements to pstats browser: do not crash on reading invalid file, and actually do a reload when executing "read" as intended. 2010-08-01 07:57:47 +00:00
Georg Brandl 3e4f2ec704 #7395: fix traceback in do_add() when no stats are loaded. Apply same fix for do_sort() and do_reverse(). 2010-08-01 07:48:43 +00:00
Raymond Hettinger c50846aaef Forward port total_ordering() and cmp_to_key(). 2010-04-05 18:56:31 +00:00
Andrew M. Kuchling 0a62823d81 Merged revisions 79603 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r79603 | andrew.kuchling | 2010-04-02 12:59:16 -0400 (Fri, 02 Apr 2010) | 1 line

  #4440: modernize a use of filter(), making it compatible with 3.x
........
2010-04-02 17:02:57 +00:00
Georg Brandl 7837a968e2 Use true kwonly arg instead of **kwds hackaround. 2009-09-02 20:33:30 +00:00
Raymond Hettinger 70b64fce96 Issue #1771: Remove cmp parameter from list.sort() and builtin.sorted(). 2008-01-30 20:15:17 +00:00
Christian Heimes e1c981161c Merged revisions 60143-60149 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r60143 | georg.brandl | 2008-01-20 15:50:05 +0100 (Sun, 20 Jan 2008) | 3 lines

  Switch mmap from old Py_FindMethod to new PyObject_GenericGetAttr attribute access.
  Fixes #1087735.
........
  r60145 | georg.brandl | 2008-01-20 20:40:58 +0100 (Sun, 20 Jan 2008) | 2 lines

  Add blurb about executable scripts on Windows. #760657.
........
  r60146 | georg.brandl | 2008-01-20 20:48:40 +0100 (Sun, 20 Jan 2008) | 2 lines

  #1219903: fix tp_richcompare docs.
........
  r60147 | georg.brandl | 2008-01-20 22:10:08 +0100 (Sun, 20 Jan 2008) | 2 lines

  Fix markup.
........
  r60148 | gregory.p.smith | 2008-01-21 08:11:11 +0100 (Mon, 21 Jan 2008) | 14 lines

  Provide a sanity check during PyThreadState_DeleteCurrent() and
  PyThreadState_Delete() to avoid an infinite loop when the tstate list
  is messed up and has somehow becomes circular and does not contain the
  current thread.

  I don't know how this happens but it does, *very* rarely.  On more than
  one hardware platform.  I have not been able to reproduce it manually.

  Attaching to a process where its happening: it has always been in an
  infinite loop over a single element tstate list that is not the tstate
  we're looking to delete.  It has been in t_bootstrap()'s call to
  PyThreadState_DeleteCurrent() as a pthread is exiting.
........
  r60149 | georg.brandl | 2008-01-21 11:24:59 +0100 (Mon, 21 Jan 2008) | 2 lines

  #1269: fix a bug in pstats.add_callers() and add a unit test file for pstats.
........
2008-01-21 11:20:28 +00:00
Georg Brandl 50da60cc5b The final occurrences of exception slicing. 2008-01-06 21:38:54 +00:00
Guido van Rossum 3172c5d263 Patch# 1258 by Christian Heimes: kill basestring.
I like this because it makes the code shorter! :-)
2007-10-16 18:12:55 +00:00
Collin Winter ce36ad8a46 Raise statement normalization in Lib/. 2007-08-30 01:19:48 +00:00
Georg Brandl a18af4e7a2 PEP 3114: rename .next() to .__next__() and add next() builtin. 2007-04-21 15:47:16 +00:00
Guido van Rossum cc2b016125 - PEP 3106: dict.iterkeys(), .iteritems(), .itervalues() are now gone;
and .keys(), .items(), .values() return dict views.

The dict views aren't fully functional yet; in particular, they can't
be compared to sets yet.  but they are useful as "iterator wells".

There are still 27 failing unit tests; I expect that many of these
have fairly trivial fixes, but there are so many, I could use help.
2007-02-11 06:12:03 +00:00
Guido van Rossum be19ed77dd Fix most trivially-findable print statements.
There's one major and one minor category still unfixed:
doctests are the major category (and I hope to be able to augment the
refactoring tool to refactor bona fide doctests soon);
other code generating print statements in strings is the minor category.

(Oh, and I don't know if the compiler package works.)
2007-02-09 05:37:30 +00:00
Guido van Rossum b940e113bf SF patch 1631942 by Collin Winter:
(a) "except E, V" -> "except E as V"
(b) V is now limited to a simple name (local variable)
(c) V is now deleted at the end of the except block
2007-01-10 16:19:56 +00:00