Issue #18874: Implement the PEP 454 (tracemalloc)

This commit is contained in:
Victor Stinner 2013-11-23 12:27:24 +01:00
parent 0fb6072fad
commit ed3b0bca3e
17 changed files with 4024 additions and 15 deletions

View File

@ -15,3 +15,4 @@ allowing you to identify bottlenecks in your programs.
profile.rst
timeit.rst
trace.rst
tracemalloc.rst

608
Doc/library/tracemalloc.rst Normal file
View File

@ -0,0 +1,608 @@
:mod:`tracemalloc` --- Trace memory allocations
===============================================
.. module:: tracemalloc
:synopsis: Trace memory allocations.
The tracemalloc module is a debug tool to trace memory blocks allocated by
Python. It provides the following information:
* Traceback where an object was allocated
* Statistics on allocated memory blocks per filename and per line number:
total size, number and average size of allocated memory blocks
* Compute the differences between two snapshots to detect memory leaks
To trace most memory blocks allocated by Python, the module should be started
as early as possible by setting the :envvar:`PYTHONTRACEMALLOC` environment
variable to ``1``, or by using :option:`-X` ``tracemalloc`` command line
option. The :func:`tracemalloc.start` function can be called at runtime to
start tracing Python memory allocations.
By default, a trace of an allocated memory block only stores the most recent
frame (1 frame). To store 25 frames at startup: set the
:envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the
:option:`-X` ``tracemalloc=25`` command line option. The
:func:`set_traceback_limit` function can be used at runtime to set the limit.
.. versionadded:: 3.4
Examples
========
Display the top 10
------------------
Display the 10 files allocating the most memory::
import tracemalloc
tracemalloc.start()
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)
Example of output of the Python test suite::
[ Top 10 ]
<frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
<frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B
/usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
/usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
/usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
/usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
<frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B
<frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B
<string>:5: size=49.7 KiB, count=148, average=344 B
/usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB
We can see that Python loaded ``4.8 MiB`` data (bytecode and constants) from
modules and that the :mod:`collections` module allocated ``244 KiB`` to build
:class:`~collections.namedtuple` types.
See :meth:`Snapshot.statistics` for more options.
Compute differences
-------------------
Take two snapshots and display the differences::
import tracemalloc
tracemalloc.start()
# ... start your application ...
snapshot1 = tracemalloc.take_snapshot()
# ... call the function leaking memory ...
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("[ Top 10 differences ]")
for stat in top_stats[:10]:
print(stat)
Example of output before/after running some tests of the Python test suite::
[ Top 10 differences ]
<frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B
/usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B
/usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B
<frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B
/usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B
/usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB
/usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B
/usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B
/usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B
/usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B
We can see that Python loaded ``4.4 MiB`` of new data (bytecode and constants)
from modules (on of total of ``8.2 MiB``) and that the :mod:`linecache` module
cached ``940 KiB`` of Python source code to format tracebacks.
If the system has little free memory, snapshots can be written on disk using
the :meth:`Snapshot.dump` method to analyze the snapshot offline. Then use the
:meth:`Snapshot.load` method reload the snapshot.
Get the traceback of a memory block
-----------------------------------
Code to display the traceback of the biggest memory block::
import linecache
import tracemalloc
tracemalloc.set_traceback_limit(25)
tracemalloc.start()
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('traceback')
# pick the biggest memory block
stat = top_stats[0]
print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
for frame in stat.traceback:
print(' File "%s", line %s' % (frame.filename, frame.lineno))
line = linecache.getline(frame.filename, frame.lineno)
line = line.strip()
if line:
print(' ' + line)
Example of output of the Python test suite (traceback limited to 25 frames)::
903 memory blocks: 870.1 KiB
File "<frozen importlib._bootstrap>", line 716
File "<frozen importlib._bootstrap>", line 1036
File "<frozen importlib._bootstrap>", line 934
File "<frozen importlib._bootstrap>", line 1068
File "<frozen importlib._bootstrap>", line 619
File "<frozen importlib._bootstrap>", line 1581
File "<frozen importlib._bootstrap>", line 1614
File "/usr/lib/python3.4/doctest.py", line 101
import pdb
File "<frozen importlib._bootstrap>", line 284
File "<frozen importlib._bootstrap>", line 938
File "<frozen importlib._bootstrap>", line 1068
File "<frozen importlib._bootstrap>", line 619
File "<frozen importlib._bootstrap>", line 1581
File "<frozen importlib._bootstrap>", line 1614
File "/usr/lib/python3.4/test/support/__init__.py", line 1728
import doctest
File "/usr/lib/python3.4/test/test_pickletools.py", line 21
support.run_doctest(pickletools)
File "/usr/lib/python3.4/test/regrtest.py", line 1276
test_runner()
File "/usr/lib/python3.4/test/regrtest.py", line 976
display_failure=not verbose)
File "/usr/lib/python3.4/test/regrtest.py", line 761
match_tests=ns.match_tests)
File "/usr/lib/python3.4/test/regrtest.py", line 1563
main()
File "/usr/lib/python3.4/test/__main__.py", line 3
regrtest.main_in_temp_cwd()
File "/usr/lib/python3.4/runpy.py", line 73
exec(code, run_globals)
File "/usr/lib/python3.4/runpy.py", line 160
"__main__", fname, loader, pkg_name)
We can see that most memory was allocated in the :mod:`importlib` module to
load data (bytecode and constants) from modules: ``870 KiB``. The traceback is
where the :mod:`importlib` loaded data for the the last time: on the ``import
pdb`` line of the :mod:`doctest` module. The traceback may change if a new
module is loaded.
Pretty top
----------
Code to display the 10 lines allocating the most memory with a pretty output,
ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files::
import os
import tracemalloc
def display_top(snapshot, group_by='lineno', limit=10):
snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
tracemalloc.Filter(False, "<unknown>"),
))
top_stats = snapshot.statistics(group_by)
print("Top %s lines" % limit)
for index, stat in enumerate(top_stats[:limit], 1):
frame = stat.traceback[0]
# replace "/path/to/module/file.py" with "module/file.py"
filename = os.sep.join(frame.filename.split(os.sep)[-2:])
print("#%s: %s:%s: %.1f KiB"
% (index, filename, frame.lineno,
stat.size / 1024))
other = top_stats[limit:]
if other:
size = sum(stat.size for stat in other)
print("%s other: %.1f KiB" % (len(other), size / 1024))
total = sum(stat.size for stat in top_stats)
print("Total allocated size: %.1f KiB" % (total / 1024))
tracemalloc.start()
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
display_top(snapshot, 10)
Example of output of the Python test suite::
2013-11-08 14:16:58.149320: Top 10 lines
#1: collections/__init__.py:368: 291.9 KiB
#2: Lib/doctest.py:1291: 200.2 KiB
#3: unittest/case.py:571: 160.3 KiB
#4: Lib/abc.py:133: 99.8 KiB
#5: urllib/parse.py:476: 71.8 KiB
#6: <string>:5: 62.7 KiB
#7: Lib/base64.py:140: 59.8 KiB
#8: Lib/_weakrefset.py:37: 51.8 KiB
#9: collections/__init__.py:362: 50.6 KiB
#10: test/test_site.py:56: 48.0 KiB
7496 other: 4161.9 KiB
Total allocated size: 5258.8 KiB
See :meth:`Snapshot.statistics` for more options.
API
===
Functions
---------
.. function:: clear_traces()
Clear traces of memory blocks allocated by Python.
See also :func:`stop`.
.. function:: get_object_traceback(obj)
Get the traceback where the Python object *obj* was allocated.
Return a :class:`Traceback` instance, or ``None`` if the :mod:`tracemalloc`
module is not tracing memory allocations or did not trace the allocation of
the object.
See also :func:`gc.get_referrers` and :func:`sys.getsizeof` functions.
.. function:: get_traceback_limit()
Get the maximum number of frames stored in the traceback of a trace.
By default, a trace of a memory block only stores the most recent
frame: the limit is ``1``.
Use the :func:`set_traceback_limit` function to change the limit.
.. function:: get_traced_memory()
Get the current size and maximum size of memory blocks traced by the
:mod:`tracemalloc` module as a tuple: ``(size: int, max_size: int)``.
.. function:: get_tracemalloc_memory()
Get the memory usage in bytes of the :mod:`tracemalloc` module used to store
traces of memory blocks.
Return an :class:`int`.
.. function:: is_tracing()
``True`` if the :mod:`tracemalloc` module is tracing Python memory
allocations, ``False`` otherwise.
See also :func:`start` and :func:`stop` functions.
.. function:: set_traceback_limit(nframe: int)
Set the maximum number of frames stored in the traceback of a trace.
*nframe* must be greater or equal to ``1``.
Storing more than ``1`` frame is only useful to compute statistics grouped
by ``'traceback'`` or to compute cumulative statistics: see the
:meth:`Snapshot.compare_to` and :meth:`Snapshot.statistics` methods.
Storing more frames increases the memory and CPU overhead of the
:mod:`tracemalloc` module. Use the :func:`get_tracemalloc_memory` function
to measure how much memory is used by the :mod:`tracemalloc` module.
The :envvar:`PYTHONTRACEMALLOC` environment variable
(``PYTHONTRACEMALLOC=NFRAME``) and the :option:`-X` ``tracemalloc=NFRAME``
command line option can be used to set the limit at startup.
Use the :func:`get_traceback_limit` function to get the current limit.
.. function:: start()
Start tracing Python memory allocations: install hooks on Python memory
allocators.
See also :func:`stop` and :func:`is_tracing` functions.
.. function:: stop()
Stop tracing Python memory allocations: uninstall hooks on Python memory
allocators. Clear also traces of memory blocks allocated by Python
Call :func:`take_snapshot` function to take a snapshot of traces before
clearing them.
See also :func:`start` and :func:`is_tracing` functions.
.. function:: take_snapshot()
Take a snapshot of traces of memory blocks allocated by Python. Return a new
:class:`Snapshot` instance.
The snapshot does not include memory blocks allocated before the
:mod:`tracemalloc` module started to trace memory allocations.
Tracebacks of traces are limited to :func:`get_traceback_limit` frames. Use
:func:`set_traceback_limit` to store more frames.
The :mod:`tracemalloc` module must be tracing memory allocations to take a
snapshot, see the the :func:`start` function.
See also the :func:`get_object_traceback` function.
Filter
------
.. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)
Filter on traces of memory blocks.
See the :func:`fnmatch.fnmatch` function for the syntax of
*filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are
replaced with ``'.py'``.
Examples:
* ``Filter(True, subprocess.__file__)`` only includes traces of the
:mod:`subprocess` module
* ``Filter(False, tracemalloc.__file__)`` excludes traces of the
:mod:`tracemalloc` module
* ``Filter(False, "<unknown>")`` excludes empty tracebacks
.. attribute:: inclusive
If *inclusive* is ``True`` (include), only trace memory blocks allocated
in a file with a name matching :attr:`filename_pattern` at line number
:attr:`lineno`.
If *inclusive* is ``False`` (exclude), ignore memory blocks allocated in
a file with a name matching :attr:`filename_pattern` at line number
:attr:`lineno`.
.. attribute:: lineno
Line number (``int``) of the filter. If *lineno* is ``None``, the filter
matches any line number.
.. attribute:: filename_pattern
Filename pattern of the filter (``str``).
.. attribute:: all_frames
If *all_frames* is ``True``, all frames of the traceback are checked. If
*all_frames* is ``False``, only the most recent frame is checked.
This attribute is ignored if the traceback limit is less than ``2``. See
the :func:`get_traceback_limit` function and
:attr:`Snapshot.traceback_limit` attribute.
Frame
-----
.. class:: Frame
Frame of a traceback.
The :class:`Traceback` class is a sequence of :class:`Frame` instances.
.. attribute:: filename
Filename (``str``).
.. attribute:: lineno
Line number (``int``).
Snapshot
--------
.. class:: Snapshot
Snapshot of traces of memory blocks allocated by Python.
The :func:`take_snapshot` function creates a snapshot instance.
.. method:: compare_to(old_snapshot: Snapshot, group_by: str, cumulative: bool=False)
Compute the differences with an old snapshot. Get statistics as a sorted
list of :class:`StatisticDiff` instances grouped by *group_by*.
See the :meth:`statistics` method for *group_by* and *cumulative*
parameters.
The result is sorted from the biggest to the smallest by: absolute value
of :attr:`StatisticDiff.size_diff`, :attr:`StatisticDiff.size`, absolute
value of :attr:`StatisticDiff.count_diff`, :attr:`Statistic.count` and
then by :attr:`StatisticDiff.traceback`.
.. method:: dump(filename)
Write the snapshot into a file.
Use :meth:`load` to reload the snapshot.
.. method:: filter_traces(filters)
Create a new :class:`Snapshot` instance with a filtered :attr:`traces`
sequence, *filters* is a list of :class:`Filter` instances. If *filters*
is an empty list, return a new :class:`Snapshot` instance with a copy of
the traces.
All inclusive filters are applied at once, a trace is ignored if no
inclusive filters match it. A trace is ignored if at least one exclusive
filter matchs it.
.. classmethod:: load(filename)
Load a snapshot from a file.
See also :meth:`dump`.
.. method:: statistics(group_by: str, cumulative: bool=False)
Get statistics as a sorted list of :class:`Statistic` instances grouped
by *group_by*:
===================== ========================
group_by description
===================== ========================
``'filename'`` filename
``'lineno'`` filename and line number
``'traceback'`` traceback
===================== ========================
If *cumulative* is ``True``, cumulate size and count of memory blocks of
all frames of the traceback of a trace, not only the most recent frame.
The cumulative mode can only be used with *group_by* equals to
``'filename'`` and ``'lineno'`` and :attr:`traceback_limit` greater than
``1``.
The result is sorted from the biggest to the smallest by:
:attr:`Statistic.size`, :attr:`Statistic.count` and then by
:attr:`Statistic.traceback`.
.. attribute:: traceback_limit
Maximum number of frames stored in the traceback of :attr:`traces`:
result of the :func:`get_traceback_limit` when the snapshot was taken.
.. attribute:: traces
Traces of all memory blocks allocated by Python: sequence of
:class:`Trace` instances.
The sequence has an undefined order. Use the :meth:`Snapshot.statistics`
method to get a sorted list of statistics.
Statistic
---------
.. class:: Statistic
Statistic on memory allocations.
:func:`Snapshot.statistics` returns a list of :class:`Statistic` instances.
See also the :class:`StatisticDiff` class.
.. attribute:: count
Number of memory blocks (``int``).
.. attribute:: size
Total size of memory blocks in bytes (``int``).
.. attribute:: traceback
Traceback where the memory block was allocated, :class:`Traceback`
instance.
StatisticDiff
-------------
.. class:: StatisticDiff
Statistic difference on memory allocations between an old and a new
:class:`Snapshot` instance.
:func:`Snapshot.compare_to` returns a list of :class:`StatisticDiff`
instances. See also the :class:`Statistic` class.
.. attribute:: count
Number of memory blocks in the new snapshot (``int``): ``0`` if
the memory blocks have been released in the new snapshot.
.. attribute:: count_diff
Difference of number of memory blocks between the old and the new
snapshots (``int``): ``0`` if the memory blocks have been allocated in
the new snapshot.
.. attribute:: size
Total size of memory blocks in bytes in the new snapshot (``int``):
``0`` if the memory blocks have been released in the new snapshot.
.. attribute:: size_diff
Difference of total size of memory blocks in bytes between the old and
the new snapshots (``int``): ``0`` if the memory blocks have been
allocated in the new snapshot.
.. attribute:: traceback
Traceback where the memory blocks were allocated, :class:`Traceback`
instance.
Trace
-----
.. class:: Trace
Trace of a memory block.
The :attr:`Snapshot.traces` attribute is a sequence of :class:`Trace`
instances.
.. attribute:: size
Size of the memory block in bytes (``int``).
.. attribute:: traceback
Traceback where the memory block was allocated, :class:`Traceback`
instance.
Traceback
---------
.. class:: Traceback
Sequence of :class:`Frame` instances sorted from the most recent frame to
the oldest frame.
A traceback contains at least ``1`` frame. If the ``tracemalloc`` module
failed to get a frame, the filename ``"<unknown>"`` at line number ``0`` is
used.
When a snapshot is taken, tracebacks of traces are limited to
:func:`get_traceback_limit` frames. See the :func:`take_snapshot` function.
The :attr:`Trace.traceback` attribute is an instance of :class:`Traceback`
instance.

View File

@ -893,3 +893,44 @@ used for the build::
Jean-loup Gailly Mark Adler
jloup@gzip.org madler@alumni.caltech.edu
cfuhash
-------
The implementtation of the hash table used by the :mod:`tracemalloc` is based
on the cfuhash project::
Copyright (c) 2005 Don Owens
All rights reserved.
This code is released under the BSD license:
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
* Neither the name of the author nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@ -376,11 +376,15 @@ Miscellaneous options
.. cmdoption:: -X
Reserved for various implementation-specific options. CPython currently
defines two possible values:
defines the following possible values:
* ``-X faulthandler`` to enable :mod:`faulthandler`;
* ``-X showrefcount`` to enable the output of the total reference count
and memory blocks (only works on debug builds);
* ``-X tracemalloc`` to enable :mod:`tracemalloc`.
* ``-X tracemalloc=NFRAME`` to enable :mod:`tracemalloc`, *NFRAME* is the
maximum number of frames stored in a trace: see the
:func:`tracemalloc.set_traceback_limit` function.
It also allows to pass arbitrary values and retrieve them through the
:data:`sys._xoptions` dictionary.
@ -392,7 +396,7 @@ Miscellaneous options
The ``-X faulthandler`` option.
.. versionadded:: 3.4
The ``-X showrefcount`` option.
The ``-X showrefcount`` and ``-X tracemalloc`` options.
Options you shouldn't use
@ -594,6 +598,16 @@ conflict.
.. versionadded:: 3.3
.. envvar:: PYTHONTRACEMALLOC
If this environment variable is set to a non-empty string, all memory
allocations made by Python are traced by the :mod:`tracemalloc` module.
The value of the variable is the maximum number of frames stored in a trace:
see the :func:`tracemalloc.set_traceback_limit` function.
.. versionadded:: 3.4
Debug-mode variables
~~~~~~~~~~~~~~~~~~~~

View File

@ -2154,3 +2154,22 @@ def patch(test_instance, object_to_patch, attr_name, new_value):
# actually override the attribute
setattr(object_to_patch, attr_name, new_value)
def run_in_subinterp(code):
"""
Run code in a subinterpreter. Raise unittest.SkipTest if the tracemalloc
module is enabled.
"""
# Issue #10915, #15751: PyGILState_*() functions don't work with
# sub-interpreters, the tracemalloc module uses these functions internally
try:
import tracemalloc
except ImportError:
pass
else:
if tracemalloc.is_tracing():
raise unittest.SkipTest("run_in_subinterp() cannot be used "
"if tracemalloc module is tracing "
"memory allocations")
return _testcapi.run_in_subinterp(code)

View File

@ -158,7 +158,7 @@ class SubinterpreterTest(unittest.TestCase):
atexit.register(f)
del atexit
"""
ret = _testcapi.run_in_subinterp(code)
ret = support.run_in_subinterp(code)
self.assertEqual(ret, 0)
self.assertEqual(atexit._ncallbacks(), n)
@ -173,7 +173,7 @@ class SubinterpreterTest(unittest.TestCase):
atexit.register(f)
atexit.__atexit = atexit
"""
ret = _testcapi.run_in_subinterp(code)
ret = support.run_in_subinterp(code)
self.assertEqual(ret, 0)
self.assertEqual(atexit._ncallbacks(), n)

View File

@ -205,7 +205,7 @@ class SubinterpreterTest(unittest.TestCase):
pickle.dump(id(builtins), f)
""".format(w)
with open(r, "rb") as f:
ret = _testcapi.run_in_subinterp(code)
ret = support.run_in_subinterp(code)
self.assertEqual(ret, 0)
self.assertNotEqual(pickle.load(f), id(sys.modules))
self.assertNotEqual(pickle.load(f), id(builtins))

View File

@ -853,7 +853,7 @@ class SubinterpThreadingTests(BaseTestCase):
os.write(%d, b"x")
threading.Thread(target=f).start()
""" % (w,)
ret = _testcapi.run_in_subinterp(code)
ret = test.support.run_in_subinterp(code)
self.assertEqual(ret, 0)
# The thread was joined properly.
self.assertEqual(os.read(r, 1), b"x")
@ -885,7 +885,7 @@ class SubinterpThreadingTests(BaseTestCase):
os.write(%d, b"x")
threading.Thread(target=f).start()
""" % (w,)
ret = _testcapi.run_in_subinterp(code)
ret = test.support.run_in_subinterp(code)
self.assertEqual(ret, 0)
# The thread was joined properly.
self.assertEqual(os.read(r, 1), b"x")

View File

@ -0,0 +1,797 @@
import _tracemalloc
import contextlib
import datetime
import os
import sys
import tracemalloc
import unittest
from unittest.mock import patch
from test.script_helper import assert_python_ok, assert_python_failure
from test import support
try:
import threading
except ImportError:
threading = None
EMPTY_STRING_SIZE = sys.getsizeof(b'')
def get_frames(nframe, lineno_delta):
frames = []
frame = sys._getframe(1)
for index in range(nframe):
code = frame.f_code
lineno = frame.f_lineno + lineno_delta
frames.append((code.co_filename, lineno))
lineno_delta = 0
frame = frame.f_back
if frame is None:
break
return tuple(frames)
def allocate_bytes(size):
nframe = tracemalloc.get_traceback_limit()
bytes_len = (size - EMPTY_STRING_SIZE)
frames = get_frames(nframe, 1)
data = b'x' * bytes_len
return data, tracemalloc.Traceback(frames)
def create_snapshots():
traceback_limit = 2
raw_traces = [
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(2, (('a.py', 5), ('b.py', 4))),
(66, (('b.py', 1),)),
(7, (('<unknown>', 0),)),
]
snapshot = tracemalloc.Snapshot(raw_traces, traceback_limit)
raw_traces2 = [
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(2, (('a.py', 5), ('b.py', 4))),
(5000, (('a.py', 5), ('b.py', 4))),
(400, (('c.py', 578),)),
]
snapshot2 = tracemalloc.Snapshot(raw_traces2, traceback_limit)
return (snapshot, snapshot2)
def frame(filename, lineno):
return tracemalloc._Frame((filename, lineno))
def traceback(*frames):
return tracemalloc.Traceback(frames)
def traceback_lineno(filename, lineno):
return traceback((filename, lineno))
def traceback_filename(filename):
return traceback_lineno(filename, 0)
class TestTracemallocEnabled(unittest.TestCase):
def setUp(self):
if tracemalloc.is_tracing():
self.skipTest("tracemalloc must be stopped before the test")
tracemalloc.set_traceback_limit(1)
tracemalloc.start()
def tearDown(self):
tracemalloc.stop()
def test_get_tracemalloc_memory(self):
data = [allocate_bytes(123) for count in range(1000)]
size = tracemalloc.get_tracemalloc_memory()
self.assertGreaterEqual(size, 0)
tracemalloc.clear_traces()
size2 = tracemalloc.get_tracemalloc_memory()
self.assertGreaterEqual(size2, 0)
self.assertLessEqual(size2, size)
def test_get_object_traceback(self):
tracemalloc.clear_traces()
obj_size = 12345
obj, obj_traceback = allocate_bytes(obj_size)
traceback = tracemalloc.get_object_traceback(obj)
self.assertEqual(traceback, obj_traceback)
def test_set_traceback_limit(self):
obj_size = 10
nframe = tracemalloc.get_traceback_limit()
self.addCleanup(tracemalloc.set_traceback_limit, nframe)
self.assertRaises(ValueError, tracemalloc.set_traceback_limit, -1)
tracemalloc.clear_traces()
tracemalloc.set_traceback_limit(10)
obj2, obj2_traceback = allocate_bytes(obj_size)
traceback = tracemalloc.get_object_traceback(obj2)
self.assertEqual(len(traceback), 10)
self.assertEqual(traceback, obj2_traceback)
tracemalloc.clear_traces()
tracemalloc.set_traceback_limit(1)
obj, obj_traceback = allocate_bytes(obj_size)
traceback = tracemalloc.get_object_traceback(obj)
self.assertEqual(len(traceback), 1)
self.assertEqual(traceback, obj_traceback)
def find_trace(self, traces, traceback):
for trace in traces:
if trace[1] == traceback._frames:
return trace
self.fail("trace not found")
def test_get_traces(self):
tracemalloc.clear_traces()
obj_size = 12345
obj, obj_traceback = allocate_bytes(obj_size)
traces = tracemalloc._get_traces()
trace = self.find_trace(traces, obj_traceback)
self.assertIsInstance(trace, tuple)
size, traceback = trace
self.assertEqual(size, obj_size)
self.assertEqual(traceback, obj_traceback._frames)
tracemalloc.stop()
self.assertEqual(tracemalloc._get_traces(), [])
def test_get_traces_intern_traceback(self):
# dummy wrappers to get more useful and identical frames in the traceback
def allocate_bytes2(size):
return allocate_bytes(size)
def allocate_bytes3(size):
return allocate_bytes2(size)
def allocate_bytes4(size):
return allocate_bytes3(size)
# Ensure that two identical tracebacks are not duplicated
tracemalloc.clear_traces()
tracemalloc.set_traceback_limit(4)
obj_size = 123
obj1, obj1_traceback = allocate_bytes4(obj_size)
obj2, obj2_traceback = allocate_bytes4(obj_size)
traces = tracemalloc._get_traces()
trace1 = self.find_trace(traces, obj1_traceback)
trace2 = self.find_trace(traces, obj2_traceback)
size1, traceback1 = trace1
size2, traceback2 = trace2
self.assertEqual(traceback2, traceback1)
self.assertIs(traceback2, traceback1)
def test_get_traced_memory(self):
# Python allocates some internals objects, so the test must tolerate
# a small difference between the expected size and the real usage
max_error = 2048
# allocate one object
obj_size = 1024 * 1024
tracemalloc.clear_traces()
obj, obj_traceback = allocate_bytes(obj_size)
size, max_size = tracemalloc.get_traced_memory()
self.assertGreaterEqual(size, obj_size)
self.assertGreaterEqual(max_size, size)
self.assertLessEqual(size - obj_size, max_error)
self.assertLessEqual(max_size - size, max_error)
# destroy the object
obj = None
size2, max_size2 = tracemalloc.get_traced_memory()
self.assertLess(size2, size)
self.assertGreaterEqual(size - size2, obj_size - max_error)
self.assertGreaterEqual(max_size2, max_size)
# clear_traces() must reset traced memory counters
tracemalloc.clear_traces()
self.assertEqual(tracemalloc.get_traced_memory(), (0, 0))
# allocate another object
obj, obj_traceback = allocate_bytes(obj_size)
size, max_size = tracemalloc.get_traced_memory()
self.assertGreater(size, 0)
# stop() rests also traced memory counters
tracemalloc.stop()
self.assertEqual(tracemalloc.get_traced_memory(), (0, 0))
def test_clear_traces(self):
obj, obj_traceback = allocate_bytes(123)
traceback = tracemalloc.get_object_traceback(obj)
self.assertIsNotNone(traceback)
tracemalloc.clear_traces()
traceback2 = tracemalloc.get_object_traceback(obj)
self.assertIsNone(traceback2)
def test_is_tracing(self):
tracemalloc.stop()
self.assertFalse(tracemalloc.is_tracing())
tracemalloc.start()
self.assertTrue(tracemalloc.is_tracing())
def test_snapshot(self):
obj, source = allocate_bytes(123)
# take a snapshot
snapshot = tracemalloc.take_snapshot()
# write on disk
snapshot.dump(support.TESTFN)
self.addCleanup(support.unlink, support.TESTFN)
# load from disk
snapshot2 = tracemalloc.Snapshot.load(support.TESTFN)
self.assertEqual(snapshot2.traces, snapshot.traces)
# tracemalloc must be tracing memory allocations to take a snapshot
tracemalloc.stop()
with self.assertRaises(RuntimeError) as cm:
tracemalloc.take_snapshot()
self.assertEqual(str(cm.exception),
"the tracemalloc module must be tracing memory "
"allocations to take a snapshot")
def test_snapshot_save_attr(self):
# take a snapshot with a new attribute
snapshot = tracemalloc.take_snapshot()
snapshot.test_attr = "new"
snapshot.dump(support.TESTFN)
self.addCleanup(support.unlink, support.TESTFN)
# load() should recreates the attribute
snapshot2 = tracemalloc.Snapshot.load(support.TESTFN)
self.assertEqual(snapshot2.test_attr, "new")
def fork_child(self):
if not tracemalloc.is_tracing():
return 2
obj_size = 12345
obj, obj_traceback = allocate_bytes(obj_size)
traceback = tracemalloc.get_object_traceback(obj)
if traceback is None:
return 3
# everything is fine
return 0
@unittest.skipUnless(hasattr(os, 'fork'), 'need os.fork()')
def test_fork(self):
# check that tracemalloc is still working after fork
pid = os.fork()
if not pid:
# child
exitcode = 1
try:
exitcode = self.fork_child()
finally:
os._exit(exitcode)
else:
pid2, status = os.waitpid(pid, 0)
self.assertTrue(os.WIFEXITED(status))
exitcode = os.WEXITSTATUS(status)
self.assertEqual(exitcode, 0)
class TestSnapshot(unittest.TestCase):
maxDiff = 4000
def test_create_snapshot(self):
raw_traces = [(5, (('a.py', 2),))]
with contextlib.ExitStack() as stack:
stack.enter_context(patch.object(tracemalloc, 'is_tracing',
return_value=True))
stack.enter_context(patch.object(tracemalloc, 'get_traceback_limit',
return_value=5))
stack.enter_context(patch.object(tracemalloc, '_get_traces',
return_value=raw_traces))
snapshot = tracemalloc.take_snapshot()
self.assertEqual(snapshot.traceback_limit, 5)
self.assertEqual(len(snapshot.traces), 1)
trace = snapshot.traces[0]
self.assertEqual(trace.size, 5)
self.assertEqual(len(trace.traceback), 1)
self.assertEqual(trace.traceback[0].filename, 'a.py')
self.assertEqual(trace.traceback[0].lineno, 2)
def test_filter_traces(self):
snapshot, snapshot2 = create_snapshots()
filter1 = tracemalloc.Filter(False, "b.py")
filter2 = tracemalloc.Filter(True, "a.py", 2)
filter3 = tracemalloc.Filter(True, "a.py", 5)
original_traces = list(snapshot.traces._traces)
# exclude b.py
snapshot3 = snapshot.filter_traces((filter1,))
self.assertEqual(snapshot3.traces._traces, [
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(2, (('a.py', 5), ('b.py', 4))),
(7, (('<unknown>', 0),)),
])
# filter_traces() must not touch the original snapshot
self.assertEqual(snapshot.traces._traces, original_traces)
# only include two lines of a.py
snapshot4 = snapshot3.filter_traces((filter2, filter3))
self.assertEqual(snapshot4.traces._traces, [
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(10, (('a.py', 2), ('b.py', 4))),
(2, (('a.py', 5), ('b.py', 4))),
])
# No filter: just duplicate the snapshot
snapshot5 = snapshot.filter_traces(())
self.assertIsNot(snapshot5, snapshot)
self.assertIsNot(snapshot5.traces, snapshot.traces)
self.assertEqual(snapshot5.traces, snapshot.traces)
def test_snapshot_group_by_line(self):
snapshot, snapshot2 = create_snapshots()
tb_0 = traceback_lineno('<unknown>', 0)
tb_a_2 = traceback_lineno('a.py', 2)
tb_a_5 = traceback_lineno('a.py', 5)
tb_b_1 = traceback_lineno('b.py', 1)
tb_c_578 = traceback_lineno('c.py', 578)
# stats per file and line
stats1 = snapshot.statistics('lineno')
self.assertEqual(stats1, [
tracemalloc.Statistic(tb_b_1, 66, 1),
tracemalloc.Statistic(tb_a_2, 30, 3),
tracemalloc.Statistic(tb_0, 7, 1),
tracemalloc.Statistic(tb_a_5, 2, 1),
])
# stats per file and line (2)
stats2 = snapshot2.statistics('lineno')
self.assertEqual(stats2, [
tracemalloc.Statistic(tb_a_5, 5002, 2),
tracemalloc.Statistic(tb_c_578, 400, 1),
tracemalloc.Statistic(tb_a_2, 30, 3),
])
# stats diff per file and line
statistics = snapshot2.compare_to(snapshot, 'lineno')
self.assertEqual(statistics, [
tracemalloc.StatisticDiff(tb_a_5, 5002, 5000, 2, 1),
tracemalloc.StatisticDiff(tb_c_578, 400, 400, 1, 1),
tracemalloc.StatisticDiff(tb_b_1, 0, -66, 0, -1),
tracemalloc.StatisticDiff(tb_0, 0, -7, 0, -1),
tracemalloc.StatisticDiff(tb_a_2, 30, 0, 3, 0),
])
def test_snapshot_group_by_file(self):
snapshot, snapshot2 = create_snapshots()
tb_0 = traceback_filename('<unknown>')
tb_a = traceback_filename('a.py')
tb_b = traceback_filename('b.py')
tb_c = traceback_filename('c.py')
# stats per file
stats1 = snapshot.statistics('filename')
self.assertEqual(stats1, [
tracemalloc.Statistic(tb_b, 66, 1),
tracemalloc.Statistic(tb_a, 32, 4),
tracemalloc.Statistic(tb_0, 7, 1),
])
# stats per file (2)
stats2 = snapshot2.statistics('filename')
self.assertEqual(stats2, [
tracemalloc.Statistic(tb_a, 5032, 5),
tracemalloc.Statistic(tb_c, 400, 1),
])
# stats diff per file
diff = snapshot2.compare_to(snapshot, 'filename')
self.assertEqual(diff, [
tracemalloc.StatisticDiff(tb_a, 5032, 5000, 5, 1),
tracemalloc.StatisticDiff(tb_c, 400, 400, 1, 1),
tracemalloc.StatisticDiff(tb_b, 0, -66, 0, -1),
tracemalloc.StatisticDiff(tb_0, 0, -7, 0, -1),
])
def test_snapshot_group_by_traceback(self):
snapshot, snapshot2 = create_snapshots()
# stats per file
tb1 = traceback(('a.py', 2), ('b.py', 4))
tb2 = traceback(('a.py', 5), ('b.py', 4))
tb3 = traceback(('b.py', 1))
tb4 = traceback(('<unknown>', 0))
stats1 = snapshot.statistics('traceback')
self.assertEqual(stats1, [
tracemalloc.Statistic(tb3, 66, 1),
tracemalloc.Statistic(tb1, 30, 3),
tracemalloc.Statistic(tb4, 7, 1),
tracemalloc.Statistic(tb2, 2, 1),
])
# stats per file (2)
tb5 = traceback(('c.py', 578))
stats2 = snapshot2.statistics('traceback')
self.assertEqual(stats2, [
tracemalloc.Statistic(tb2, 5002, 2),
tracemalloc.Statistic(tb5, 400, 1),
tracemalloc.Statistic(tb1, 30, 3),
])
# stats diff per file
diff = snapshot2.compare_to(snapshot, 'traceback')
self.assertEqual(diff, [
tracemalloc.StatisticDiff(tb2, 5002, 5000, 2, 1),
tracemalloc.StatisticDiff(tb5, 400, 400, 1, 1),
tracemalloc.StatisticDiff(tb3, 0, -66, 0, -1),
tracemalloc.StatisticDiff(tb4, 0, -7, 0, -1),
tracemalloc.StatisticDiff(tb1, 30, 0, 3, 0),
])
self.assertRaises(ValueError,
snapshot.statistics, 'traceback', cumulative=True)
def test_snapshot_group_by_cumulative(self):
snapshot, snapshot2 = create_snapshots()
tb_0 = traceback_filename('<unknown>')
tb_a = traceback_filename('a.py')
tb_b = traceback_filename('b.py')
tb_a_2 = traceback_lineno('a.py', 2)
tb_a_5 = traceback_lineno('a.py', 5)
tb_b_1 = traceback_lineno('b.py', 1)
tb_b_4 = traceback_lineno('b.py', 4)
# per file
stats = snapshot.statistics('filename', True)
self.assertEqual(stats, [
tracemalloc.Statistic(tb_b, 98, 5),
tracemalloc.Statistic(tb_a, 32, 4),
tracemalloc.Statistic(tb_0, 7, 1),
])
# per line
stats = snapshot.statistics('lineno', True)
self.assertEqual(stats, [
tracemalloc.Statistic(tb_b_1, 66, 1),
tracemalloc.Statistic(tb_b_4, 32, 4),
tracemalloc.Statistic(tb_a_2, 30, 3),
tracemalloc.Statistic(tb_0, 7, 1),
tracemalloc.Statistic(tb_a_5, 2, 1),
])
def test_trace_format(self):
snapshot, snapshot2 = create_snapshots()
trace = snapshot.traces[0]
self.assertEqual(str(trace), 'a.py:2: 10 B')
traceback = trace.traceback
self.assertEqual(str(traceback), 'a.py:2')
frame = traceback[0]
self.assertEqual(str(frame), 'a.py:2')
def test_statistic_format(self):
snapshot, snapshot2 = create_snapshots()
stats = snapshot.statistics('lineno')
stat = stats[0]
self.assertEqual(str(stat),
'b.py:1: size=66 B, count=1, average=66 B')
def test_statistic_diff_format(self):
snapshot, snapshot2 = create_snapshots()
stats = snapshot2.compare_to(snapshot, 'lineno')
stat = stats[0]
self.assertEqual(str(stat),
'a.py:5: size=5002 B (+5000 B), count=2 (+1), average=2501 B')
class TestFilters(unittest.TestCase):
maxDiff = 2048
def test_filter_attributes(self):
# test default values
f = tracemalloc.Filter(True, "abc")
self.assertEqual(f.inclusive, True)
self.assertEqual(f.filename_pattern, "abc")
self.assertIsNone(f.lineno)
self.assertEqual(f.all_frames, False)
# test custom values
f = tracemalloc.Filter(False, "test.py", 123, True)
self.assertEqual(f.inclusive, False)
self.assertEqual(f.filename_pattern, "test.py")
self.assertEqual(f.lineno, 123)
self.assertEqual(f.all_frames, True)
# parameters passed by keyword
f = tracemalloc.Filter(inclusive=False, filename_pattern="test.py", lineno=123, all_frames=True)
self.assertEqual(f.inclusive, False)
self.assertEqual(f.filename_pattern, "test.py")
self.assertEqual(f.lineno, 123)
self.assertEqual(f.all_frames, True)
# read-only attribute
self.assertRaises(AttributeError, setattr, f, "filename_pattern", "abc")
def test_filter_match(self):
# filter without line number
f = tracemalloc.Filter(True, "abc")
self.assertTrue(f._match_frame("abc", 0))
self.assertTrue(f._match_frame("abc", 5))
self.assertTrue(f._match_frame("abc", 10))
self.assertFalse(f._match_frame("12356", 0))
self.assertFalse(f._match_frame("12356", 5))
self.assertFalse(f._match_frame("12356", 10))
f = tracemalloc.Filter(False, "abc")
self.assertFalse(f._match_frame("abc", 0))
self.assertFalse(f._match_frame("abc", 5))
self.assertFalse(f._match_frame("abc", 10))
self.assertTrue(f._match_frame("12356", 0))
self.assertTrue(f._match_frame("12356", 5))
self.assertTrue(f._match_frame("12356", 10))
# filter with line number > 0
f = tracemalloc.Filter(True, "abc", 5)
self.assertFalse(f._match_frame("abc", 0))
self.assertTrue(f._match_frame("abc", 5))
self.assertFalse(f._match_frame("abc", 10))
self.assertFalse(f._match_frame("12356", 0))
self.assertFalse(f._match_frame("12356", 5))
self.assertFalse(f._match_frame("12356", 10))
f = tracemalloc.Filter(False, "abc", 5)
self.assertTrue(f._match_frame("abc", 0))
self.assertFalse(f._match_frame("abc", 5))
self.assertTrue(f._match_frame("abc", 10))
self.assertTrue(f._match_frame("12356", 0))
self.assertTrue(f._match_frame("12356", 5))
self.assertTrue(f._match_frame("12356", 10))
# filter with line number 0
f = tracemalloc.Filter(True, "abc", 0)
self.assertTrue(f._match_frame("abc", 0))
self.assertFalse(f._match_frame("abc", 5))
self.assertFalse(f._match_frame("abc", 10))
self.assertFalse(f._match_frame("12356", 0))
self.assertFalse(f._match_frame("12356", 5))
self.assertFalse(f._match_frame("12356", 10))
f = tracemalloc.Filter(False, "abc", 0)
self.assertFalse(f._match_frame("abc", 0))
self.assertTrue(f._match_frame("abc", 5))
self.assertTrue(f._match_frame("abc", 10))
self.assertTrue(f._match_frame("12356", 0))
self.assertTrue(f._match_frame("12356", 5))
self.assertTrue(f._match_frame("12356", 10))
def test_filter_match_filename(self):
def fnmatch(inclusive, filename, pattern):
f = tracemalloc.Filter(inclusive, pattern)
return f._match_frame(filename, 0)
self.assertTrue(fnmatch(True, "abc", "abc"))
self.assertFalse(fnmatch(True, "12356", "abc"))
self.assertFalse(fnmatch(True, "<unknown>", "abc"))
self.assertFalse(fnmatch(False, "abc", "abc"))
self.assertTrue(fnmatch(False, "12356", "abc"))
self.assertTrue(fnmatch(False, "<unknown>", "abc"))
def test_filter_match_filename_joker(self):
def fnmatch(filename, pattern):
filter = tracemalloc.Filter(True, pattern)
return filter._match_frame(filename, 0)
# empty string
self.assertFalse(fnmatch('abc', ''))
self.assertFalse(fnmatch('', 'abc'))
self.assertTrue(fnmatch('', ''))
self.assertTrue(fnmatch('', '*'))
# no *
self.assertTrue(fnmatch('abc', 'abc'))
self.assertFalse(fnmatch('abc', 'abcd'))
self.assertFalse(fnmatch('abc', 'def'))
# a*
self.assertTrue(fnmatch('abc', 'a*'))
self.assertTrue(fnmatch('abc', 'abc*'))
self.assertFalse(fnmatch('abc', 'b*'))
self.assertFalse(fnmatch('abc', 'abcd*'))
# a*b
self.assertTrue(fnmatch('abc', 'a*c'))
self.assertTrue(fnmatch('abcdcx', 'a*cx'))
self.assertFalse(fnmatch('abb', 'a*c'))
self.assertFalse(fnmatch('abcdce', 'a*cx'))
# a*b*c
self.assertTrue(fnmatch('abcde', 'a*c*e'))
self.assertTrue(fnmatch('abcbdefeg', 'a*bd*eg'))
self.assertFalse(fnmatch('abcdd', 'a*c*e'))
self.assertFalse(fnmatch('abcbdefef', 'a*bd*eg'))
# replace .pyc and .pyo suffix with .py
self.assertTrue(fnmatch('a.pyc', 'a.py'))
self.assertTrue(fnmatch('a.pyo', 'a.py'))
self.assertTrue(fnmatch('a.py', 'a.pyc'))
self.assertTrue(fnmatch('a.py', 'a.pyo'))
if os.name == 'nt':
# case insensitive
self.assertTrue(fnmatch('aBC', 'ABc'))
self.assertTrue(fnmatch('aBcDe', 'Ab*dE'))
self.assertTrue(fnmatch('a.pyc', 'a.PY'))
self.assertTrue(fnmatch('a.PYO', 'a.py'))
self.assertTrue(fnmatch('a.py', 'a.PYC'))
self.assertTrue(fnmatch('a.PY', 'a.pyo'))
else:
# case sensitive
self.assertFalse(fnmatch('aBC', 'ABc'))
self.assertFalse(fnmatch('aBcDe', 'Ab*dE'))
self.assertFalse(fnmatch('a.pyc', 'a.PY'))
self.assertFalse(fnmatch('a.PYO', 'a.py'))
self.assertFalse(fnmatch('a.py', 'a.PYC'))
self.assertFalse(fnmatch('a.PY', 'a.pyo'))
if os.name == 'nt':
# normalize alternate separator "/" to the standard separator "\"
self.assertTrue(fnmatch(r'a/b', r'a\b'))
self.assertTrue(fnmatch(r'a\b', r'a/b'))
self.assertTrue(fnmatch(r'a/b\c', r'a\b/c'))
self.assertTrue(fnmatch(r'a/b/c', r'a\b\c'))
else:
# there is no alternate separator
self.assertFalse(fnmatch(r'a/b', r'a\b'))
self.assertFalse(fnmatch(r'a\b', r'a/b'))
self.assertFalse(fnmatch(r'a/b\c', r'a\b/c'))
self.assertFalse(fnmatch(r'a/b/c', r'a\b\c'))
def test_filter_match_trace(self):
t1 = (("a.py", 2), ("b.py", 3))
t2 = (("b.py", 4), ("b.py", 5))
t3 = (("c.py", 5), ('<unknown>', 0))
unknown = (('<unknown>', 0),)
f = tracemalloc.Filter(True, "b.py", all_frames=True)
self.assertTrue(f._match_traceback(t1))
self.assertTrue(f._match_traceback(t2))
self.assertFalse(f._match_traceback(t3))
self.assertFalse(f._match_traceback(unknown))
f = tracemalloc.Filter(True, "b.py", all_frames=False)
self.assertFalse(f._match_traceback(t1))
self.assertTrue(f._match_traceback(t2))
self.assertFalse(f._match_traceback(t3))
self.assertFalse(f._match_traceback(unknown))
f = tracemalloc.Filter(False, "b.py", all_frames=True)
self.assertFalse(f._match_traceback(t1))
self.assertFalse(f._match_traceback(t2))
self.assertTrue(f._match_traceback(t3))
self.assertTrue(f._match_traceback(unknown))
f = tracemalloc.Filter(False, "b.py", all_frames=False)
self.assertTrue(f._match_traceback(t1))
self.assertFalse(f._match_traceback(t2))
self.assertTrue(f._match_traceback(t3))
self.assertTrue(f._match_traceback(unknown))
f = tracemalloc.Filter(False, "<unknown>", all_frames=False)
self.assertTrue(f._match_traceback(t1))
self.assertTrue(f._match_traceback(t2))
self.assertTrue(f._match_traceback(t3))
self.assertFalse(f._match_traceback(unknown))
f = tracemalloc.Filter(True, "<unknown>", all_frames=True)
self.assertFalse(f._match_traceback(t1))
self.assertFalse(f._match_traceback(t2))
self.assertTrue(f._match_traceback(t3))
self.assertTrue(f._match_traceback(unknown))
f = tracemalloc.Filter(False, "<unknown>", all_frames=True)
self.assertTrue(f._match_traceback(t1))
self.assertTrue(f._match_traceback(t2))
self.assertFalse(f._match_traceback(t3))
self.assertFalse(f._match_traceback(unknown))
class TestCommandLine(unittest.TestCase):
def test_env_var(self):
# not tracing by default
code = 'import tracemalloc; print(tracemalloc.is_tracing())'
ok, stdout, stderr = assert_python_ok('-c', code)
stdout = stdout.rstrip()
self.assertEqual(stdout, b'False')
# PYTHON* environment varibles must be ignored when -E option is
# present
code = 'import tracemalloc; print(tracemalloc.is_tracing())'
ok, stdout, stderr = assert_python_ok('-E', '-c', code, PYTHONTRACEMALLOC='1')
stdout = stdout.rstrip()
self.assertEqual(stdout, b'False')
# tracing at startup
code = 'import tracemalloc; print(tracemalloc.is_tracing())'
ok, stdout, stderr = assert_python_ok('-c', code, PYTHONTRACEMALLOC='1')
stdout = stdout.rstrip()
self.assertEqual(stdout, b'True')
# start and set the number of frames
code = 'import tracemalloc; print(tracemalloc.get_traceback_limit())'
ok, stdout, stderr = assert_python_ok('-c', code, PYTHONTRACEMALLOC='10')
stdout = stdout.rstrip()
self.assertEqual(stdout, b'10')
def test_env_var_invalid(self):
for nframe in (-1, 0, 5000):
with self.subTest(nframe=nframe):
with support.SuppressCrashReport():
ok, stdout, stderr = assert_python_failure(
'-c', 'pass',
PYTHONTRACEMALLOC=str(nframe))
self.assertIn(b'PYTHONTRACEMALLOC must be an integer '
b'in range [1; 100]',
stderr)
def test_sys_xoptions(self):
for xoptions, nframe in (
('tracemalloc', 1),
('tracemalloc=1', 1),
('tracemalloc=15', 15),
):
with self.subTest(xoptions=xoptions, nframe=nframe):
code = 'import tracemalloc; print(tracemalloc.get_traceback_limit())'
ok, stdout, stderr = assert_python_ok('-X', xoptions, '-c', code)
stdout = stdout.rstrip()
self.assertEqual(stdout, str(nframe).encode('ascii'))
def test_sys_xoptions_invalid(self):
for nframe in (-1, 0, 5000):
with self.subTest(nframe=nframe):
with support.SuppressCrashReport():
args = ('-X', 'tracemalloc=%s' % nframe, '-c', 'pass')
ok, stdout, stderr = assert_python_failure(*args)
self.assertIn(b'-X tracemalloc=NFRAME: number of frame must '
b'be an integer in range [1; 100]',
stderr)
def test_main():
support.run_unittest(
TestTracemallocEnabled,
TestSnapshot,
TestFilters,
TestCommandLine,
)
if __name__ == "__main__":
test_main()

464
Lib/tracemalloc.py Normal file
View File

@ -0,0 +1,464 @@
from collections import Sequence
from functools import total_ordering
import fnmatch
import os.path
import pickle
# Import types and functions implemented in C
from _tracemalloc import *
from _tracemalloc import _get_object_traceback, _get_traces
def _format_size(size, sign):
for unit in ('B', 'KiB', 'MiB', 'GiB', 'TiB'):
if abs(size) < 100 and unit != 'B':
# 3 digits (xx.x UNIT)
if sign:
return "%+.1f %s" % (size, unit)
else:
return "%.1f %s" % (size, unit)
if abs(size) < 10 * 1024 or unit == 'TiB':
# 4 or 5 digits (xxxx UNIT)
if sign:
return "%+.0f %s" % (size, unit)
else:
return "%.0f %s" % (size, unit)
size /= 1024
class Statistic:
"""
Statistic difference on memory allocations between two Snapshot instance.
"""
__slots__ = ('traceback', 'size', 'count')
def __init__(self, traceback, size, count):
self.traceback = traceback
self.size = size
self.count = count
def __hash__(self):
return (self.traceback, self.size, self.count)
def __eq__(self, other):
return (self.traceback == other.traceback
and self.size == other.size
and self.count == other.count)
def __str__(self):
text = ("%s: size=%s, count=%i"
% (self.traceback,
_format_size(self.size, False),
self.count))
if self.count:
average = self.size / self.count
text += ", average=%s" % _format_size(average, False)
return text
def __repr__(self):
return ('<Statistic traceback=%r size=%i count=%i>'
% (self.traceback, self.size, self.count))
def _sort_key(self):
return (self.size, self.count, self.traceback)
class StatisticDiff:
"""
Statistic difference on memory allocations between an old and a new
Snapshot instance.
"""
__slots__ = ('traceback', 'size', 'size_diff', 'count', 'count_diff')
def __init__(self, traceback, size, size_diff, count, count_diff):
self.traceback = traceback
self.size = size
self.size_diff = size_diff
self.count = count
self.count_diff = count_diff
def __hash__(self):
return (self.traceback, self.size, self.size_diff,
self.count, self.count_diff)
def __eq__(self, other):
return (self.traceback == other.traceback
and self.size == other.size
and self.size_diff == other.size_diff
and self.count == other.count
and self.count_diff == other.count_diff)
def __str__(self):
text = ("%s: size=%s (%s), count=%i (%+i)"
% (self.traceback,
_format_size(self.size, False),
_format_size(self.size_diff, True),
self.count,
self.count_diff))
if self.count:
average = self.size / self.count
text += ", average=%s" % _format_size(average, False)
return text
def __repr__(self):
return ('<StatisticDiff traceback=%r size=%i (%+i) count=%i (%+i)>'
% (self.traceback, self.size, self.size_diff,
self.count, self.count_diff))
def _sort_key(self):
return (abs(self.size_diff), self.size,
abs(self.count_diff), self.count,
self.traceback)
def _compare_grouped_stats(old_group, new_group):
statistics = []
for traceback, stat in new_group.items():
previous = old_group.pop(traceback, None)
if previous is not None:
stat = StatisticDiff(traceback,
stat.size, stat.size - previous.size,
stat.count, stat.count - previous.count)
else:
stat = StatisticDiff(traceback,
stat.size, stat.size,
stat.count, stat.count)
statistics.append(stat)
for traceback, stat in old_group.items():
stat = StatisticDiff(traceback, 0, -stat.size, 0, -stat.count)
statistics.append(stat)
return statistics
@total_ordering
class Frame:
"""
Frame of a traceback.
"""
__slots__ = ("_frame",)
def __init__(self, frame):
self._frame = frame
@property
def filename(self):
return self._frame[0]
@property
def lineno(self):
return self._frame[1]
def __eq__(self, other):
return (self._frame == other._frame)
def __lt__(self, other):
return (self._frame < other._frame)
def __hash__(self):
return hash(self._frame)
def __str__(self):
return "%s:%s" % (self.filename, self.lineno)
def __repr__(self):
return "<Frame filename=%r lineno=%r>" % (self.filename, self.lineno)
@total_ordering
class Traceback(Sequence):
"""
Sequence of Frame instances sorted from the most recent frame
to the oldest frame.
"""
__slots__ = ("_frames",)
def __init__(self, frames):
Sequence.__init__(self)
self._frames = frames
def __len__(self):
return len(self._frames)
def __getitem__(self, index):
trace = self._frames[index]
return Frame(trace)
def __contains__(self, frame):
return frame._frame in self._frames
def __hash__(self):
return hash(self._frames)
def __eq__(self, other):
return (self._frames == other._frames)
def __lt__(self, other):
return (self._frames < other._frames)
def __str__(self):
return str(self[0])
def __repr__(self):
return "<Traceback %r>" % (tuple(self),)
def get_object_traceback(obj):
"""
Get the traceback where the Python object *obj* was allocated.
Return a Traceback instance.
Return None if the tracemalloc module is not tracing memory allocations or
did not trace the allocation of the object.
"""
frames = _get_object_traceback(obj)
if frames is not None:
return Traceback(frames)
else:
return None
class Trace:
"""
Trace of a memory block.
"""
__slots__ = ("_trace",)
def __init__(self, trace):
self._trace = trace
@property
def size(self):
return self._trace[0]
@property
def traceback(self):
return Traceback(self._trace[1])
def __eq__(self, other):
return (self._trace == other._trace)
def __hash__(self):
return hash(self._trace)
def __str__(self):
return "%s: %s" % (self.traceback, _format_size(self.size, False))
def __repr__(self):
return ("<Trace size=%s, traceback=%r>"
% (_format_size(self.size, False), self.traceback))
class _Traces(Sequence):
def __init__(self, traces):
Sequence.__init__(self)
self._traces = traces
def __len__(self):
return len(self._traces)
def __getitem__(self, index):
trace = self._traces[index]
return Trace(trace)
def __contains__(self, trace):
return trace._trace in self._traces
def __eq__(self, other):
return (self._traces == other._traces)
def __repr__(self):
return "<Traces len=%s>" % len(self)
def _normalize_filename(filename):
filename = os.path.normcase(filename)
if filename.endswith(('.pyc', '.pyo')):
filename = filename[:-1]
return filename
class Filter:
def __init__(self, inclusive, filename_pattern,
lineno=None, all_frames=False):
self.inclusive = inclusive
self._filename_pattern = _normalize_filename(filename_pattern)
self.lineno = lineno
self.all_frames = all_frames
@property
def filename_pattern(self):
return self._filename_pattern
def __match_frame(self, filename, lineno):
filename = _normalize_filename(filename)
if not fnmatch.fnmatch(filename, self._filename_pattern):
return False
if self.lineno is None:
return True
else:
return (lineno == self.lineno)
def _match_frame(self, filename, lineno):
return self.__match_frame(filename, lineno) ^ (not self.inclusive)
def _match_traceback(self, traceback):
if self.all_frames:
if any(self.__match_frame(filename, lineno)
for filename, lineno in traceback):
return self.inclusive
else:
return (not self.inclusive)
else:
filename, lineno = traceback[0]
return self._match_frame(filename, lineno)
class Snapshot:
"""
Snapshot of traces of memory blocks allocated by Python.
"""
def __init__(self, traces, traceback_limit):
self.traces = _Traces(traces)
self.traceback_limit = traceback_limit
def dump(self, filename):
"""
Write the snapshot into a file.
"""
with open(filename, "wb") as fp:
pickle.dump(self, fp, pickle.HIGHEST_PROTOCOL)
@staticmethod
def load(filename):
"""
Load a snapshot from a file.
"""
with open(filename, "rb") as fp:
return pickle.load(fp)
def _filter_trace(self, include_filters, exclude_filters, trace):
traceback = trace[1]
if include_filters:
if not any(trace_filter._match_traceback(traceback)
for trace_filter in include_filters):
return False
if exclude_filters:
if any(not trace_filter._match_traceback(traceback)
for trace_filter in exclude_filters):
return False
return True
def filter_traces(self, filters):
"""
Create a new Snapshot instance with a filtered traces sequence, filters
is a list of Filter instances. If filters is an empty list, return a
new Snapshot instance with a copy of the traces.
"""
if filters:
include_filters = []
exclude_filters = []
for trace_filter in filters:
if trace_filter.inclusive:
include_filters.append(trace_filter)
else:
exclude_filters.append(trace_filter)
new_traces = [trace for trace in self.traces._traces
if self._filter_trace(include_filters,
exclude_filters,
trace)]
else:
new_traces = self.traces._traces.copy()
return Snapshot(new_traces, self.traceback_limit)
def _group_by(self, key_type, cumulative):
if key_type not in ('traceback', 'filename', 'lineno'):
raise ValueError("unknown key_type: %r" % (key_type,))
if cumulative and key_type not in ('lineno', 'filename'):
raise ValueError("cumulative mode cannot by used "
"with key type %r" % key_type)
if cumulative and self.traceback_limit < 2:
raise ValueError("cumulative mode needs tracebacks with at least "
"2 frames, traceback limit is %s"
% self.traceback_limit)
stats = {}
tracebacks = {}
if not cumulative:
for trace in self.traces._traces:
size, trace_traceback = trace
try:
traceback = tracebacks[trace_traceback]
except KeyError:
if key_type == 'traceback':
frames = trace_traceback
elif key_type == 'lineno':
frames = trace_traceback[:1]
else: # key_type == 'filename':
frames = ((trace_traceback[0][0], 0),)
traceback = Traceback(frames)
tracebacks[trace_traceback] = traceback
try:
stat = stats[traceback]
stat.size += size
stat.count += 1
except KeyError:
stats[traceback] = Statistic(traceback, size, 1)
else:
# cumulative statistics
for trace in self.traces._traces:
size, trace_traceback = trace
for frame in trace_traceback:
try:
traceback = tracebacks[frame]
except KeyError:
if key_type == 'lineno':
frames = (frame,)
else: # key_type == 'filename':
frames = ((frame[0], 0),)
traceback = Traceback(frames)
tracebacks[frame] = traceback
try:
stat = stats[traceback]
stat.size += size
stat.count += 1
except KeyError:
stats[traceback] = Statistic(traceback, size, 1)
return stats
def statistics(self, key_type, cumulative=False):
"""
Group statistics by key_type. Return a sorted list of Statistic
instances.
"""
grouped = self._group_by(key_type, cumulative)
statistics = list(grouped.values())
statistics.sort(reverse=True, key=Statistic._sort_key)
return statistics
def compare_to(self, old_snapshot, key_type, cumulative=False):
"""
Compute the differences with an old snapshot old_snapshot. Get
statistics as a sorted list of StatisticDiff instances, grouped by
group_by.
"""
new_group = self._group_by(key_type, cumulative)
old_group = old_snapshot._group_by(key_type, cumulative)
statistics = _compare_grouped_stats(old_group, new_group)
statistics.sort(reverse=True, key=StatisticDiff._sort_key)
return statistics
def take_snapshot():
"""
Take a snapshot of traces of memory blocks allocated by Python.
"""
if not is_tracing():
raise RuntimeError("the tracemalloc module must be tracing memory "
"allocations to take a snapshot")
traces = _get_traces()
traceback_limit = get_traceback_limit()
return Snapshot(traces, traceback_limit)

View File

@ -102,7 +102,7 @@ PYTHONPATH=$(COREPYTHONPATH)
# various reasons; therefore they are listed here instead of in the
# normal order.
# This only contains the minimal set of modules required to run the
# This only contains the minimal set of modules required to run the
# setup.py script in the root of the Python source tree.
posix posixmodule.c # posix (UNIX) system calls
@ -115,7 +115,7 @@ _weakref _weakref.c # weak references
_functools _functoolsmodule.c # Tools for working with functions and callable objects
_operator _operator.c # operator.add() and similar goodies
_collections _collectionsmodule.c # Container types
itertools itertoolsmodule.c # Functions creating iterators for efficient looping
itertools itertoolsmodule.c # Functions creating iterators for efficient looping
atexit atexitmodule.c # Register functions to be run at interpreter-shutdown
_stat _stat.c # stat.h interface
@ -132,12 +132,15 @@ zipimport zipimport.c
# faulthandler module
faulthandler faulthandler.c
# debug tool to trace memory blocks allocated by Python
_tracemalloc _tracemalloc.c hashtable.c
# The rest of the modules listed in this file are all commented out by
# default. Usually they can be detected and built as dynamically
# loaded modules by the new setup.py script added in Python 2.1. If
# you're on a platform that doesn't support dynamic loading, want to
# compile modules statically into the Python binary, or need to
# specify some odd set of compiler switches, you can uncomment the
# you're on a platform that doesn't support dynamic loading, want to
# compile modules statically into the Python binary, or need to
# specify some odd set of compiler switches, you can uncomment the
# appropriate lines below.
# ======================================================================
@ -186,7 +189,7 @@ _symtable symtablemodule.c
# supported...)
#fcntl fcntlmodule.c # fcntl(2) and ioctl(2)
#spwd spwdmodule.c # spwd(3)
#spwd spwdmodule.c # spwd(3)
#grp grpmodule.c # grp(3)
#select selectmodule.c # select(2); not on ancient System V
@ -302,7 +305,7 @@ _symtable symtablemodule.c
#_curses _cursesmodule.c -lcurses -ltermcap
# Wrapper for the panel library that's part of ncurses and SYSV curses.
#_curses_panel _curses_panel.c -lpanel -lncurses
#_curses_panel _curses_panel.c -lpanel -lncurses
# Modules that provide persistent dictionary-like semantics. You will

1407
Modules/_tracemalloc.c Normal file

File diff suppressed because it is too large Load Diff

518
Modules/hashtable.c Normal file
View File

@ -0,0 +1,518 @@
/* The implementation of the hash table (_Py_hashtable_t) is based on the cfuhash
project:
http://sourceforge.net/projects/libcfu/
Copyright of cfuhash:
----------------------------------
Creation date: 2005-06-24 21:22:40
Authors: Don
Change log:
Copyright (c) 2005 Don Owens
All rights reserved.
This code is released under the BSD license:
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
* Neither the name of the author nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.
----------------------------------
*/
#include "Python.h"
#include "hashtable.h"
#define HASHTABLE_MIN_SIZE 16
#define HASHTABLE_HIGH 0.50
#define HASHTABLE_LOW 0.10
#define HASHTABLE_REHASH_FACTOR 2.0 / (HASHTABLE_LOW + HASHTABLE_HIGH)
#define BUCKETS_HEAD(SLIST) \
((_Py_hashtable_entry_t *)_Py_SLIST_HEAD(&(SLIST)))
#define TABLE_HEAD(HT, BUCKET) \
((_Py_hashtable_entry_t *)_Py_SLIST_HEAD(&(HT)->buckets[BUCKET]))
#define ENTRY_NEXT(ENTRY) \
((_Py_hashtable_entry_t *)_Py_SLIST_ITEM_NEXT(ENTRY))
#define HASHTABLE_ITEM_SIZE(HT) \
(sizeof(_Py_hashtable_entry_t) + (HT)->data_size)
/* Forward declaration */
static void hashtable_rehash(_Py_hashtable_t *ht);
static void
_Py_slist_init(_Py_slist_t *list)
{
list->head = NULL;
}
static void
_Py_slist_prepend(_Py_slist_t *list, _Py_slist_item_t *item)
{
item->next = list->head;
list->head = item;
}
static void
_Py_slist_remove(_Py_slist_t *list, _Py_slist_item_t *previous,
_Py_slist_item_t *item)
{
if (previous != NULL)
previous->next = item->next;
else
list->head = item->next;
}
Py_uhash_t
_Py_hashtable_hash_int(const void *key)
{
return (Py_uhash_t)key;
}
Py_uhash_t
_Py_hashtable_hash_ptr(const void *key)
{
return (Py_uhash_t)_Py_HashPointer((void *)key);
}
int
_Py_hashtable_compare_direct(const void *key, const _Py_hashtable_entry_t *entry)
{
return entry->key == key;
}
/* makes sure the real size of the buckets array is a power of 2 */
static size_t
round_size(size_t s)
{
size_t i;
if (s < HASHTABLE_MIN_SIZE)
return HASHTABLE_MIN_SIZE;
i = 1;
while (i < s)
i <<= 1;
return i;
}
_Py_hashtable_t *
_Py_hashtable_new_full(size_t data_size, size_t init_size,
_Py_hashtable_hash_func hash_func,
_Py_hashtable_compare_func compare_func,
_Py_hashtable_copy_data_func copy_data_func,
_Py_hashtable_free_data_func free_data_func,
_Py_hashtable_get_data_size_func get_data_size_func,
_Py_hashtable_allocator_t *allocator)
{
_Py_hashtable_t *ht;
size_t buckets_size;
_Py_hashtable_allocator_t alloc;
if (allocator == NULL) {
alloc.malloc = PyMem_RawMalloc;
alloc.free = PyMem_RawFree;
}
else
alloc = *allocator;
ht = (_Py_hashtable_t *)alloc.malloc(sizeof(_Py_hashtable_t));
if (ht == NULL)
return ht;
ht->num_buckets = round_size(init_size);
ht->entries = 0;
ht->data_size = data_size;
buckets_size = ht->num_buckets * sizeof(ht->buckets[0]);
ht->buckets = alloc.malloc(buckets_size);
if (ht->buckets == NULL) {
alloc.free(ht);
return NULL;
}
memset(ht->buckets, 0, buckets_size);
ht->hash_func = hash_func;
ht->compare_func = compare_func;
ht->copy_data_func = copy_data_func;
ht->free_data_func = free_data_func;
ht->get_data_size_func = get_data_size_func;
ht->alloc = alloc;
return ht;
}
_Py_hashtable_t *
_Py_hashtable_new(size_t data_size,
_Py_hashtable_hash_func hash_func,
_Py_hashtable_compare_func compare_func)
{
return _Py_hashtable_new_full(data_size, HASHTABLE_MIN_SIZE,
hash_func, compare_func,
NULL, NULL, NULL, NULL);
}
size_t
_Py_hashtable_size(_Py_hashtable_t *ht)
{
size_t size;
size_t hv;
size = sizeof(_Py_hashtable_t);
/* buckets */
size += ht->num_buckets * sizeof(_Py_hashtable_entry_t *);
/* entries */
size += ht->entries * HASHTABLE_ITEM_SIZE(ht);
/* data linked from entries */
if (ht->get_data_size_func) {
for (hv = 0; hv < ht->num_buckets; hv++) {
_Py_hashtable_entry_t *entry;
for (entry = TABLE_HEAD(ht, hv); entry; entry = ENTRY_NEXT(entry)) {
void *data;
data = _Py_HASHTABLE_ENTRY_DATA_AS_VOID_P(entry);
size += ht->get_data_size_func(data);
}
}
}
return size;
}
#ifdef Py_DEBUG
void
_Py_hashtable_print_stats(_Py_hashtable_t *ht)
{
size_t size;
size_t chain_len, max_chain_len, total_chain_len, nchains;
_Py_hashtable_entry_t *entry;
size_t hv;
double load;
size = _Py_hashtable_size(ht);
load = (double)ht->entries / ht->num_buckets;
max_chain_len = 0;
total_chain_len = 0;
nchains = 0;
for (hv = 0; hv < ht->num_buckets; hv++) {
entry = TABLE_HEAD(ht, hv);
if (entry != NULL) {
chain_len = 0;
for (; entry; entry = ENTRY_NEXT(entry)) {
chain_len++;
}
if (chain_len > max_chain_len)
max_chain_len = chain_len;
total_chain_len += chain_len;
nchains++;
}
}
printf("hash table %p: entries=%zu/%zu (%.0f%%), ",
ht, ht->entries, ht->num_buckets, load * 100.0);
if (nchains)
printf("avg_chain_len=%.1f, ", (double)total_chain_len / nchains);
printf("max_chain_len=%zu, %zu kB\n",
max_chain_len, size / 1024);
}
#endif
/* Get an entry. Return NULL if the key does not exist. */
_Py_hashtable_entry_t *
_Py_hashtable_get_entry(_Py_hashtable_t *ht, const void *key)
{
Py_uhash_t key_hash;
size_t index;
_Py_hashtable_entry_t *entry;
key_hash = ht->hash_func(key);
index = key_hash & (ht->num_buckets - 1);
for (entry = TABLE_HEAD(ht, index); entry != NULL; entry = ENTRY_NEXT(entry)) {
if (entry->key_hash == key_hash && ht->compare_func(key, entry))
break;
}
return entry;
}
static int
_hashtable_pop_entry(_Py_hashtable_t *ht, const void *key, void *data, size_t data_size)
{
Py_uhash_t key_hash;
size_t index;
_Py_hashtable_entry_t *entry, *previous;
key_hash = ht->hash_func(key);
index = key_hash & (ht->num_buckets - 1);
previous = NULL;
for (entry = TABLE_HEAD(ht, index); entry != NULL; entry = ENTRY_NEXT(entry)) {
if (entry->key_hash == key_hash && ht->compare_func(key, entry))
break;
previous = entry;
}
if (entry == NULL)
return 0;
_Py_slist_remove(&ht->buckets[index], (_Py_slist_item_t *)previous,
(_Py_slist_item_t *)entry);
ht->entries--;
if (data != NULL)
_Py_HASHTABLE_ENTRY_READ_DATA(ht, data, data_size, entry);
ht->alloc.free(entry);
if ((float)ht->entries / (float)ht->num_buckets < HASHTABLE_LOW)
hashtable_rehash(ht);
return 1;
}
/* Add a new entry to the hash. The key must not be present in the hash table.
Return 0 on success, -1 on memory error. */
int
_Py_hashtable_set(_Py_hashtable_t *ht, const void *key,
void *data, size_t data_size)
{
Py_uhash_t key_hash;
size_t index;
_Py_hashtable_entry_t *entry;
assert(data != NULL || data_size == 0);
#ifndef NDEBUG
/* Don't write the assertion on a single line because it is interesting
to know the duplicated entry if the assertion failed. The entry can
be read using a debugger. */
entry = _Py_hashtable_get_entry(ht, key);
assert(entry == NULL);
#endif
key_hash = ht->hash_func(key);
index = key_hash & (ht->num_buckets - 1);
entry = ht->alloc.malloc(HASHTABLE_ITEM_SIZE(ht));
if (entry == NULL) {
/* memory allocation failed */
return -1;
}
entry->key = (void *)key;
entry->key_hash = key_hash;
assert(data_size == ht->data_size);
memcpy(_PY_HASHTABLE_ENTRY_DATA(entry), data, data_size);
_Py_slist_prepend(&ht->buckets[index], (_Py_slist_item_t*)entry);
ht->entries++;
if ((float)ht->entries / (float)ht->num_buckets > HASHTABLE_HIGH)
hashtable_rehash(ht);
return 0;
}
/* Get data from an entry. Copy entry data into data and return 1 if the entry
exists, return 0 if the entry does not exist. */
int
_Py_hashtable_get(_Py_hashtable_t *ht, const void *key, void *data, size_t data_size)
{
_Py_hashtable_entry_t *entry;
assert(data != NULL);
entry = _Py_hashtable_get_entry(ht, key);
if (entry == NULL)
return 0;
_Py_HASHTABLE_ENTRY_READ_DATA(ht, data, data_size, entry);
return 1;
}
int
_Py_hashtable_pop(_Py_hashtable_t *ht, const void *key, void *data, size_t data_size)
{
assert(data != NULL);
assert(ht->free_data_func == NULL);
return _hashtable_pop_entry(ht, key, data, data_size);
}
/* Delete an entry. The entry must exist. */
void
_Py_hashtable_delete(_Py_hashtable_t *ht, const void *key)
{
#ifndef NDEBUG
int found = _hashtable_pop_entry(ht, key, NULL, 0);
assert(found);
#else
(void)_hashtable_pop_entry(ht, key, NULL, 0);
#endif
}
/* Prototype for a pointer to a function to be called foreach
key/value pair in the hash by hashtable_foreach(). Iteration
stops if a non-zero value is returned. */
int
_Py_hashtable_foreach(_Py_hashtable_t *ht,
int (*func) (_Py_hashtable_entry_t *entry, void *arg),
void *arg)
{
_Py_hashtable_entry_t *entry;
size_t hv;
for (hv = 0; hv < ht->num_buckets; hv++) {
for (entry = TABLE_HEAD(ht, hv); entry; entry = ENTRY_NEXT(entry)) {
int res = func(entry, arg);
if (res)
return res;
}
}
return 0;
}
static void
hashtable_rehash(_Py_hashtable_t *ht)
{
size_t buckets_size, new_size, bucket;
_Py_slist_t *old_buckets = NULL;
size_t old_num_buckets;
new_size = round_size((size_t)(ht->entries * HASHTABLE_REHASH_FACTOR));
if (new_size == ht->num_buckets)
return;
old_num_buckets = ht->num_buckets;
buckets_size = new_size * sizeof(ht->buckets[0]);
old_buckets = ht->buckets;
ht->buckets = ht->alloc.malloc(buckets_size);
if (ht->buckets == NULL) {
/* cancel rehash on memory allocation failure */
ht->buckets = old_buckets ;
/* memory allocation failed */
return;
}
memset(ht->buckets, 0, buckets_size);
ht->num_buckets = new_size;
for (bucket = 0; bucket < old_num_buckets; bucket++) {
_Py_hashtable_entry_t *entry, *next;
for (entry = BUCKETS_HEAD(old_buckets[bucket]); entry != NULL; entry = next) {
size_t entry_index;
assert(ht->hash_func(entry->key) == entry->key_hash);
next = ENTRY_NEXT(entry);
entry_index = entry->key_hash & (new_size - 1);
_Py_slist_prepend(&ht->buckets[entry_index], (_Py_slist_item_t*)entry);
}
}
ht->alloc.free(old_buckets);
}
void
_Py_hashtable_clear(_Py_hashtable_t *ht)
{
_Py_hashtable_entry_t *entry, *next;
size_t i;
for (i=0; i < ht->num_buckets; i++) {
for (entry = TABLE_HEAD(ht, i); entry != NULL; entry = next) {
next = ENTRY_NEXT(entry);
if (ht->free_data_func)
ht->free_data_func(_Py_HASHTABLE_ENTRY_DATA_AS_VOID_P(entry));
ht->alloc.free(entry);
}
_Py_slist_init(&ht->buckets[i]);
}
ht->entries = 0;
hashtable_rehash(ht);
}
void
_Py_hashtable_destroy(_Py_hashtable_t *ht)
{
size_t i;
for (i = 0; i < ht->num_buckets; i++) {
_Py_slist_item_t *entry = ht->buckets[i].head;
while (entry) {
_Py_slist_item_t *entry_next = entry->next;
if (ht->free_data_func)
ht->free_data_func(_Py_HASHTABLE_ENTRY_DATA_AS_VOID_P(entry));
ht->alloc.free(entry);
entry = entry_next;
}
}
ht->alloc.free(ht->buckets);
ht->alloc.free(ht);
}
/* Return a copy of the hash table */
_Py_hashtable_t *
_Py_hashtable_copy(_Py_hashtable_t *src)
{
_Py_hashtable_t *dst;
_Py_hashtable_entry_t *entry;
size_t bucket;
int err;
void *data, *new_data;
dst = _Py_hashtable_new_full(src->data_size, src->num_buckets,
src->hash_func, src->compare_func,
src->copy_data_func, src->free_data_func,
src->get_data_size_func, &src->alloc);
if (dst == NULL)
return NULL;
for (bucket=0; bucket < src->num_buckets; bucket++) {
entry = TABLE_HEAD(src, bucket);
for (; entry; entry = ENTRY_NEXT(entry)) {
if (src->copy_data_func) {
data = _Py_HASHTABLE_ENTRY_DATA_AS_VOID_P(entry);
new_data = src->copy_data_func(data);
if (new_data != NULL)
err = _Py_hashtable_set(dst, entry->key,
&new_data, src->data_size);
else
err = 1;
}
else {
data = _PY_HASHTABLE_ENTRY_DATA(entry);
err = _Py_hashtable_set(dst, entry->key, data, src->data_size);
}
if (err) {
_Py_hashtable_destroy(dst);
return NULL;
}
}
}
return dst;
}

128
Modules/hashtable.h Normal file
View File

@ -0,0 +1,128 @@
#ifndef Py_HASHTABLE_H
#define Py_HASHTABLE_H
/* The whole API is private */
#ifndef Py_LIMITED_API
typedef struct _Py_slist_item_s {
struct _Py_slist_item_s *next;
} _Py_slist_item_t;
typedef struct {
_Py_slist_item_t *head;
} _Py_slist_t;
#define _Py_SLIST_ITEM_NEXT(ITEM) (((_Py_slist_item_t *)ITEM)->next)
#define _Py_SLIST_HEAD(SLIST) (((_Py_slist_t *)SLIST)->head)
typedef struct {
/* used by _Py_hashtable_t.buckets to link entries */
_Py_slist_item_t _Py_slist_item;
const void *key;
Py_uhash_t key_hash;
/* data follows */
} _Py_hashtable_entry_t;
#define _PY_HASHTABLE_ENTRY_DATA(ENTRY) \
((char *)(ENTRY) + sizeof(_Py_hashtable_entry_t))
#define _Py_HASHTABLE_ENTRY_DATA_AS_VOID_P(ENTRY) \
(*(void **)_PY_HASHTABLE_ENTRY_DATA(ENTRY))
#define _Py_HASHTABLE_ENTRY_READ_DATA(TABLE, DATA, DATA_SIZE, ENTRY) \
do { \
assert((DATA_SIZE) == (TABLE)->data_size); \
memcpy(DATA, _PY_HASHTABLE_ENTRY_DATA(ENTRY), DATA_SIZE); \
} while (0)
typedef Py_uhash_t (*_Py_hashtable_hash_func) (const void *key);
typedef int (*_Py_hashtable_compare_func) (const void *key, const _Py_hashtable_entry_t *he);
typedef void* (*_Py_hashtable_copy_data_func)(void *data);
typedef void (*_Py_hashtable_free_data_func)(void *data);
typedef size_t (*_Py_hashtable_get_data_size_func)(void *data);
typedef struct {
/* allocate a memory block */
void* (*malloc) (size_t size);
/* release a memory block */
void (*free) (void *ptr);
} _Py_hashtable_allocator_t;
typedef struct {
size_t num_buckets;
size_t entries; /* Total number of entries in the table. */
_Py_slist_t *buckets;
size_t data_size;
_Py_hashtable_hash_func hash_func;
_Py_hashtable_compare_func compare_func;
_Py_hashtable_copy_data_func copy_data_func;
_Py_hashtable_free_data_func free_data_func;
_Py_hashtable_get_data_size_func get_data_size_func;
_Py_hashtable_allocator_t alloc;
} _Py_hashtable_t;
/* hash and compare functions for integers and pointers */
PyAPI_FUNC(Py_uhash_t) _Py_hashtable_hash_ptr(const void *key);
PyAPI_FUNC(Py_uhash_t) _Py_hashtable_hash_int(const void *key);
PyAPI_FUNC(int) _Py_hashtable_compare_direct(const void *key, const _Py_hashtable_entry_t *entry);
PyAPI_FUNC(_Py_hashtable_t *) _Py_hashtable_new(
size_t data_size,
_Py_hashtable_hash_func hash_func,
_Py_hashtable_compare_func compare_func);
PyAPI_FUNC(_Py_hashtable_t *) _Py_hashtable_new_full(
size_t data_size,
size_t init_size,
_Py_hashtable_hash_func hash_func,
_Py_hashtable_compare_func compare_func,
_Py_hashtable_copy_data_func copy_data_func,
_Py_hashtable_free_data_func free_data_func,
_Py_hashtable_get_data_size_func get_data_size_func,
_Py_hashtable_allocator_t *allocator);
PyAPI_FUNC(_Py_hashtable_t *) _Py_hashtable_copy(_Py_hashtable_t *src);
PyAPI_FUNC(void) _Py_hashtable_clear(_Py_hashtable_t *ht);
PyAPI_FUNC(void) _Py_hashtable_destroy(_Py_hashtable_t *ht);
typedef int (*_Py_hashtable_foreach_func) (_Py_hashtable_entry_t *entry, void *arg);
PyAPI_FUNC(int) _Py_hashtable_foreach(
_Py_hashtable_t *ht,
_Py_hashtable_foreach_func func, void *arg);
PyAPI_FUNC(size_t) _Py_hashtable_size(_Py_hashtable_t *ht);
PyAPI_FUNC(_Py_hashtable_entry_t*) _Py_hashtable_get_entry(
_Py_hashtable_t *ht,
const void *key);
PyAPI_FUNC(int) _Py_hashtable_set(
_Py_hashtable_t *ht,
const void *key,
void *data,
size_t data_size);
PyAPI_FUNC(int) _Py_hashtable_get(
_Py_hashtable_t *ht,
const void *key,
void *data,
size_t data_size);
PyAPI_FUNC(int) _Py_hashtable_pop(
_Py_hashtable_t *ht,
const void *key,
void *data,
size_t data_size);
PyAPI_FUNC(void) _Py_hashtable_delete(
_Py_hashtable_t *ht,
const void *key);
#define _Py_HASHTABLE_SET(TABLE, KEY, DATA) \
_Py_hashtable_set(TABLE, KEY, &(DATA), sizeof(DATA))
#define _Py_HASHTABLE_GET(TABLE, KEY, DATA) \
_Py_hashtable_get(TABLE, KEY, &(DATA), sizeof(DATA))
#endif /* Py_LIMITED_API */
#endif

View File

@ -13,6 +13,7 @@ extern PyObject* PyInit_binascii(void);
extern PyObject* PyInit_cmath(void);
extern PyObject* PyInit_errno(void);
extern PyObject* PyInit_faulthandler(void);
extern PyObject* PyInit__tracemalloc(void);
extern PyObject* PyInit_gc(void);
extern PyObject* PyInit_math(void);
extern PyObject* PyInit__md5(void);
@ -102,6 +103,7 @@ struct _inittab _PyImport_Inittab[] = {
{"msvcrt", PyInit_msvcrt},
{"_locale", PyInit__locale},
#endif
{"_tracemalloc", PyInit__tracemalloc},
/* XXX Should _winapi go in a WIN32 block? not WIN64? */
{"_winapi", PyInit__winapi},

View File

@ -449,6 +449,7 @@
<ClInclude Include="..\Include\unicodeobject.h" />
<ClInclude Include="..\Include\weakrefobject.h" />
<ClInclude Include="..\Modules\_math.h" />
<ClInclude Include="..\Modules\hashtable.h" />
<ClInclude Include="..\Modules\rotatingtree.h" />
<ClInclude Include="..\Modules\sre.h" />
<ClInclude Include="..\Modules\sre_constants.h" />
@ -517,6 +518,7 @@
<ClCompile Include="..\Modules\errnomodule.c" />
<ClCompile Include="..\Modules\faulthandler.c" />
<ClCompile Include="..\Modules\gcmodule.c" />
<ClCompile Include="..\Modules\hashtable.c" />
<ClCompile Include="..\Modules\itertoolsmodule.c" />
<ClCompile Include="..\Modules\main.c" />
<ClCompile Include="..\Modules\mathmodule.c" />
@ -532,6 +534,7 @@
<ClCompile Include="..\Modules\signalmodule.c" />
<ClCompile Include="..\Modules\symtablemodule.c" />
<ClCompile Include="..\Modules\_threadmodule.c" />
<ClCompile Include="..\Modules\_tracemalloc.c" />
<ClCompile Include="..\Modules\timemodule.c" />
<ClCompile Include="..\Modules\xxsubtype.c" />
<ClCompile Include="..\Modules\zipimport.c" />
@ -684,4 +687,4 @@
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>
</Project>

View File

@ -105,6 +105,7 @@ extern void PyLong_Fini(void);
extern int _PyFaulthandler_Init(void);
extern void _PyFaulthandler_Fini(void);
extern void _PyHash_Fini(void);
extern int _PyTraceMalloc_Init(void);
#ifdef WITH_THREAD
extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *);
@ -454,6 +455,9 @@ _Py_InitializeEx_Private(int install_sigs, int install_importlib)
if (install_sigs)
initsigs(); /* Signal handling stuff, including initintr() */
if (_PyTraceMalloc_Init() < 0)
Py_FatalError("Py_Initialize: can't initialize tracemalloc");
initmain(interp); /* Module __main__ */
if (initstdio() < 0)
Py_FatalError(