2006-04-19 12:27:33 -03:00
|
|
|
________________________________________________________________________
|
|
|
|
|
|
|
|
|
|
PYBENCH - A Python Benchmark Suite
|
|
|
|
|
________________________________________________________________________
|
|
|
|
|
|
|
|
|
|
Extendable suite of of low-level benchmarks for measuring
|
|
|
|
|
the performance of the Python implementation
|
|
|
|
|
(interpreter, compiler or VM).
|
|
|
|
|
|
|
|
|
|
pybench is a collection of tests that provides a standardized way to
|
|
|
|
|
measure the performance of Python implementations. It takes a very
|
|
|
|
|
close look at different aspects of Python programs and let's you
|
|
|
|
|
decide which factors are more important to you than others, rather
|
|
|
|
|
than wrapping everything up in one number, like the other performance
|
|
|
|
|
tests do (e.g. pystone which is included in the Python Standard
|
|
|
|
|
Library).
|
|
|
|
|
|
|
|
|
|
pybench has been used in the past by several Python developers to
|
|
|
|
|
track down performance bottlenecks or to demonstrate the impact of
|
|
|
|
|
optimizations and new features in Python.
|
|
|
|
|
|
|
|
|
|
The command line interface for pybench is the file pybench.py. Run
|
|
|
|
|
this script with option '--help' to get a listing of the possible
|
|
|
|
|
options. Without options, pybench will simply execute the benchmark
|
|
|
|
|
and then print out a report to stdout.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Micro-Manual
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
|
|
Run 'pybench.py -h' to see the help screen.
|
|
|
|
|
Run 'pybench.py' to just let the benchmark suite do it's thing and
|
|
|
|
|
'pybench.py -f <file>' to have it store the results in a file too.
|
|
|
|
|
|
|
|
|
|
This is the current output of pybench.py --help:
|
|
|
|
|
|
|
|
|
|
Synopsis:
|
|
|
|
|
pybench.py [option] files...
|
|
|
|
|
|
|
|
|
|
Options and default settings:
|
|
|
|
|
-n arg number of rounds (10)
|
|
|
|
|
-f arg save benchmark to file arg ()
|
|
|
|
|
-c arg compare benchmark with the one in file arg ()
|
|
|
|
|
-s arg show benchmark in file arg, then exit ()
|
|
|
|
|
-S show statistics of benchmarks (0)
|
|
|
|
|
-w arg set warp factor to arg (20)
|
|
|
|
|
-d hide noise in compares (0)
|
|
|
|
|
--no-gc disable garbage collection (0)
|
2006-05-26 19:33:20 -03:00
|
|
|
--no-syscheck "disable" sys check interval (set to sys.maxint) (0)
|
|
|
|
|
-t arg tests containing substring ()
|
|
|
|
|
-C arg number of calibration runs (20)
|
2006-04-19 12:27:33 -03:00
|
|
|
-v generate verbose output
|
|
|
|
|
-h show this help text
|
|
|
|
|
--help show this help text
|
|
|
|
|
--debug enable debugging
|
|
|
|
|
--copyright show copyright
|
|
|
|
|
--examples show examples of usage
|
|
|
|
|
|
|
|
|
|
Version:
|
|
|
|
|
1.3
|
|
|
|
|
|
|
|
|
|
The normal operation is to run the suite and display the
|
|
|
|
|
results. Use -f to save them for later reuse or comparisms.
|
|
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
|
|
|
|
|
python1.5 pybench.py -w 100 -f p15
|
|
|
|
|
python1.4 pybench.py -w 100 -f p14
|
|
|
|
|
python pybench.py -s p15 -c p14
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
License
|
|
|
|
|
-------
|
|
|
|
|
|
|
|
|
|
See LICENSE file.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Sample output
|
|
|
|
|
-------------
|
|
|
|
|
|
|
|
|
|
PYBENCH 1.3
|
|
|
|
|
|
|
|
|
|
Machine Details:
|
|
|
|
|
Platform ID: Linux-2.6.8-24.19-default-x86_64-with-SuSE-9.2-x86-64
|
|
|
|
|
Executable: /home/lemburg/projects/Python/Installation/bin/python
|
|
|
|
|
Python: 2.5a1.0
|
|
|
|
|
Compiler: GCC 3.3.4 (pre 3.3.5 20040809)
|
|
|
|
|
Build: Apr 9 2006 01:50:57 (#trunk)
|
|
|
|
|
|
|
|
|
|
Searching for tests...
|
|
|
|
|
BuiltinFunctionCalls
|
|
|
|
|
BuiltinMethodLookup
|
|
|
|
|
CompareFloats
|
|
|
|
|
CompareFloatsIntegers
|
|
|
|
|
CompareIntegers
|
|
|
|
|
CompareInternedStrings
|
|
|
|
|
CompareLongs
|
|
|
|
|
CompareStrings
|
|
|
|
|
CompareUnicode
|
|
|
|
|
ConcatStrings
|
|
|
|
|
ConcatUnicode
|
|
|
|
|
CreateInstances
|
|
|
|
|
CreateStringsWithConcat
|
|
|
|
|
CreateUnicodeWithConcat
|
|
|
|
|
DictCreation
|
|
|
|
|
DictWithFloatKeys
|
|
|
|
|
DictWithIntegerKeys
|
|
|
|
|
DictWithStringKeys
|
|
|
|
|
ForLoops
|
|
|
|
|
IfThenElse
|
|
|
|
|
ListSlicing
|
|
|
|
|
NestedForLoops
|
|
|
|
|
NormalClassAttribute
|
|
|
|
|
NormalInstanceAttribute
|
|
|
|
|
PythonFunctionCalls
|
|
|
|
|
PythonMethodCalls
|
|
|
|
|
Recursion
|
|
|
|
|
SecondImport
|
|
|
|
|
SecondPackageImport
|
|
|
|
|
SecondSubmoduleImport
|
|
|
|
|
SimpleComplexArithmetic
|
|
|
|
|
SimpleDictManipulation
|
|
|
|
|
SimpleFloatArithmetic
|
|
|
|
|
SimpleIntFloatArithmetic
|
|
|
|
|
SimpleIntegerArithmetic
|
|
|
|
|
SimpleListManipulation
|
|
|
|
|
SimpleLongArithmetic
|
|
|
|
|
SmallLists
|
|
|
|
|
SmallTuples
|
|
|
|
|
SpecialClassAttribute
|
|
|
|
|
SpecialInstanceAttribute
|
|
|
|
|
StringMappings
|
|
|
|
|
StringPredicates
|
|
|
|
|
StringSlicing
|
|
|
|
|
TryExcept
|
|
|
|
|
TryRaiseExcept
|
|
|
|
|
TupleSlicing
|
|
|
|
|
UnicodeMappings
|
|
|
|
|
UnicodePredicates
|
|
|
|
|
UnicodeProperties
|
|
|
|
|
UnicodeSlicing
|
|
|
|
|
|
|
|
|
|
Running 10 round(s) of the suite:
|
|
|
|
|
|
|
|
|
|
...
|
|
|
|
|
|
|
|
|
|
Round 10 real abs overhead
|
|
|
|
|
BuiltinFunctionCalls: 0.030r 0.030a 0.000o
|
|
|
|
|
BuiltinMethodLookup: 0.059r 0.060a 0.001o
|
|
|
|
|
CompareFloats: 0.050r 0.050a 0.000o
|
|
|
|
|
CompareFloatsIntegers: 0.050r 0.050a 0.000o
|
|
|
|
|
CompareIntegers: 0.070r 0.070a 0.000o
|
|
|
|
|
CompareInternedStrings: 0.039r 0.040a 0.001o
|
|
|
|
|
CompareLongs: 0.050r 0.050a 0.000o
|
|
|
|
|
CompareStrings: 0.060r 0.060a 0.000o
|
|
|
|
|
CompareUnicode: 0.060r 0.060a 0.000o
|
|
|
|
|
ConcatStrings: 0.040r 0.040a 0.000o
|
|
|
|
|
ConcatUnicode: 0.050r 0.050a 0.000o
|
|
|
|
|
CreateInstances: 0.050r 0.050a 0.000o
|
|
|
|
|
CreateStringsWithConcat: 0.029r 0.030a 0.001o
|
|
|
|
|
CreateUnicodeWithConcat: 0.060r 0.060a 0.000o
|
|
|
|
|
DictCreation: 0.040r 0.040a 0.000o
|
|
|
|
|
DictWithFloatKeys: 0.089r 0.090a 0.000o
|
|
|
|
|
DictWithIntegerKeys: 0.059r 0.060a 0.001o
|
|
|
|
|
DictWithStringKeys: 0.070r 0.070a 0.001o
|
|
|
|
|
ForLoops: 0.050r 0.050a 0.000o
|
|
|
|
|
IfThenElse: 0.070r 0.070a 0.000o
|
|
|
|
|
ListSlicing: 0.030r 0.030a 0.000o
|
|
|
|
|
NestedForLoops: 0.030r 0.030a 0.000o
|
|
|
|
|
NormalClassAttribute: 0.060r 0.060a 0.000o
|
|
|
|
|
NormalInstanceAttribute: 0.060r 0.060a 0.000o
|
|
|
|
|
PythonFunctionCalls: 0.060r 0.060a 0.000o
|
|
|
|
|
PythonMethodCalls: 0.050r 0.050a 0.000o
|
|
|
|
|
Recursion: 0.050r 0.050a 0.000o
|
|
|
|
|
SecondImport: 0.030r 0.030a 0.000o
|
|
|
|
|
SecondPackageImport: 0.030r 0.030a 0.000o
|
|
|
|
|
SecondSubmoduleImport: 0.040r 0.040a 0.000o
|
|
|
|
|
SimpleComplexArithmetic: 0.030r 0.030a 0.000o
|
|
|
|
|
SimpleDictManipulation: 0.040r 0.040a 0.000o
|
|
|
|
|
SimpleFloatArithmetic: 0.050r 0.050a 0.001o
|
|
|
|
|
SimpleIntFloatArithmetic: 0.060r 0.060a 0.000o
|
|
|
|
|
SimpleIntegerArithmetic: 0.060r 0.060a 0.000o
|
|
|
|
|
SimpleListManipulation: 0.030r 0.030a 0.000o
|
|
|
|
|
SimpleLongArithmetic: 0.030r 0.030a 0.000o
|
|
|
|
|
SmallLists: 0.050r 0.050a 0.000o
|
|
|
|
|
SmallTuples: 0.050r 0.050a 0.000o
|
|
|
|
|
SpecialClassAttribute: 0.060r 0.060a 0.000o
|
|
|
|
|
SpecialInstanceAttribute: 0.079r 0.080a 0.001o
|
|
|
|
|
StringMappings: 0.060r 0.060a 0.000o
|
|
|
|
|
StringPredicates: 0.049r 0.050a 0.001o
|
|
|
|
|
StringSlicing: 0.039r 0.040a 0.000o
|
|
|
|
|
TryExcept: 0.079r 0.080a 0.001o
|
|
|
|
|
TryRaiseExcept: 0.059r 0.060a 0.001o
|
|
|
|
|
TupleSlicing: 0.050r 0.050a 0.000o
|
|
|
|
|
UnicodeMappings: 0.070r 0.070a 0.001o
|
|
|
|
|
UnicodePredicates: 0.059r 0.060a 0.001o
|
|
|
|
|
UnicodeProperties: 0.059r 0.060a 0.001o
|
|
|
|
|
UnicodeSlicing: 0.050r 0.050a 0.000o
|
|
|
|
|
----------------------
|
|
|
|
|
Average round time: 2.937 seconds
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Tests: per run per oper. overhead
|
|
|
|
|
------------------------------------------------------------------------
|
|
|
|
|
BuiltinFunctionCalls: 29.85 ms 0.23 us 0.00 ms
|
|
|
|
|
BuiltinMethodLookup: 66.85 ms 0.13 us 0.50 ms
|
|
|
|
|
CompareFloats: 43.00 ms 0.10 us 0.00 ms
|
|
|
|
|
CompareFloatsIntegers: 51.80 ms 0.12 us 0.00 ms
|
|
|
|
|
CompareIntegers: 70.70 ms 0.08 us 0.50 ms
|
|
|
|
|
CompareInternedStrings: 41.40 ms 0.08 us 0.50 ms
|
|
|
|
|
CompareLongs: 47.90 ms 0.11 us 0.00 ms
|
|
|
|
|
CompareStrings: 58.50 ms 0.12 us 0.50 ms
|
|
|
|
|
CompareUnicode: 56.55 ms 0.15 us 0.50 ms
|
|
|
|
|
ConcatStrings: 44.75 ms 0.30 us 0.00 ms
|
|
|
|
|
ConcatUnicode: 54.55 ms 0.36 us 0.50 ms
|
|
|
|
|
CreateInstances: 50.95 ms 1.21 us 0.00 ms
|
|
|
|
|
CreateStringsWithConcat: 28.85 ms 0.14 us 0.50 ms
|
|
|
|
|
CreateUnicodeWithConcat: 53.75 ms 0.27 us 0.00 ms
|
|
|
|
|
DictCreation: 41.90 ms 0.28 us 0.00 ms
|
|
|
|
|
DictWithFloatKeys: 88.50 ms 0.15 us 0.50 ms
|
|
|
|
|
DictWithIntegerKeys: 62.55 ms 0.10 us 0.50 ms
|
|
|
|
|
DictWithStringKeys: 60.50 ms 0.10 us 0.50 ms
|
|
|
|
|
ForLoops: 46.90 ms 4.69 us 0.00 ms
|
|
|
|
|
IfThenElse: 60.55 ms 0.09 us 0.00 ms
|
|
|
|
|
ListSlicing: 29.90 ms 8.54 us 0.00 ms
|
|
|
|
|
NestedForLoops: 33.95 ms 0.10 us 0.00 ms
|
|
|
|
|
NormalClassAttribute: 62.75 ms 0.10 us 0.50 ms
|
|
|
|
|
NormalInstanceAttribute: 61.80 ms 0.10 us 0.50 ms
|
|
|
|
|
PythonFunctionCalls: 60.00 ms 0.36 us 0.00 ms
|
|
|
|
|
PythonMethodCalls: 50.00 ms 0.67 us 0.00 ms
|
|
|
|
|
Recursion: 46.85 ms 3.75 us 0.00 ms
|
|
|
|
|
SecondImport: 35.00 ms 1.40 us 0.00 ms
|
|
|
|
|
SecondPackageImport: 32.00 ms 1.28 us 0.00 ms
|
|
|
|
|
SecondSubmoduleImport: 38.00 ms 1.52 us 0.00 ms
|
|
|
|
|
SimpleComplexArithmetic: 26.85 ms 0.12 us 0.00 ms
|
|
|
|
|
SimpleDictManipulation: 40.85 ms 0.14 us 0.00 ms
|
|
|
|
|
SimpleFloatArithmetic: 48.70 ms 0.09 us 0.50 ms
|
|
|
|
|
SimpleIntFloatArithmetic: 57.70 ms 0.09 us 0.00 ms
|
|
|
|
|
SimpleIntegerArithmetic: 58.75 ms 0.09 us 0.50 ms
|
|
|
|
|
SimpleListManipulation: 34.80 ms 0.13 us 0.00 ms
|
|
|
|
|
SimpleLongArithmetic: 30.95 ms 0.19 us 0.50 ms
|
|
|
|
|
SmallLists: 47.60 ms 0.19 us 0.00 ms
|
|
|
|
|
SmallTuples: 48.80 ms 0.20 us 0.50 ms
|
|
|
|
|
SpecialClassAttribute: 61.70 ms 0.10 us 0.00 ms
|
|
|
|
|
SpecialInstanceAttribute: 76.70 ms 0.13 us 0.50 ms
|
|
|
|
|
StringMappings: 58.70 ms 0.47 us 0.00 ms
|
|
|
|
|
StringPredicates: 50.00 ms 0.18 us 1.00 ms
|
|
|
|
|
StringSlicing: 39.65 ms 0.23 us 0.50 ms
|
|
|
|
|
TryExcept: 84.45 ms 0.06 us 0.50 ms
|
|
|
|
|
TryRaiseExcept: 61.75 ms 4.12 us 0.50 ms
|
|
|
|
|
TupleSlicing: 48.95 ms 0.47 us 0.00 ms
|
|
|
|
|
UnicodeMappings: 71.50 ms 3.97 us 0.50 ms
|
|
|
|
|
UnicodePredicates: 52.75 ms 0.23 us 1.00 ms
|
|
|
|
|
UnicodeProperties: 61.90 ms 0.31 us 1.00 ms
|
|
|
|
|
UnicodeSlicing: 53.75 ms 0.31 us 0.50 ms
|
|
|
|
|
------------------------------------------------------------------------
|
|
|
|
|
Average round time: 2937.00 ms
|
|
|
|
|
|
|
|
|
|
________________________________________________________________________
|
|
|
|
|
|
|
|
|
|
Writing New Tests
|
|
|
|
|
________________________________________________________________________
|
|
|
|
|
|
|
|
|
|
pybench tests are simple modules defining one or more pybench.Test
|
|
|
|
|
subclasses.
|
|
|
|
|
|
|
|
|
|
Writing a test essentially boils down to providing two methods:
|
|
|
|
|
.test() which runs .rounds number of .operations test operations each
|
|
|
|
|
and .calibrate() which does the same except that it doesn't actually
|
|
|
|
|
execute the operations.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Here's an example:
|
|
|
|
|
------------------
|
|
|
|
|
|
|
|
|
|
from pybench import Test
|
|
|
|
|
|
|
|
|
|
class IntegerCounting(Test):
|
|
|
|
|
|
|
|
|
|
# Version number of the test as float (x.yy); this is important
|
|
|
|
|
# for comparisons of benchmark runs - tests with unequal version
|
|
|
|
|
# number will not get compared.
|
|
|
|
|
version = 1.0
|
|
|
|
|
|
|
|
|
|
# The number of abstract operations done in each round of the
|
|
|
|
|
# test. An operation is the basic unit of what you want to
|
|
|
|
|
# measure. The benchmark will output the amount of run-time per
|
|
|
|
|
# operation. Note that in order to raise the measured timings
|
|
|
|
|
# significantly above noise level, it is often required to repeat
|
|
|
|
|
# sets of operations more than once per test round. The measured
|
|
|
|
|
# overhead per test round should be less than 1 second.
|
|
|
|
|
operations = 20
|
|
|
|
|
|
|
|
|
|
# Number of rounds to execute per test run. This should be
|
|
|
|
|
# adjusted to a figure that results in a test run-time of between
|
|
|
|
|
# 20-50 seconds.
|
|
|
|
|
rounds = 100000
|
|
|
|
|
|
|
|
|
|
def test(self):
|
|
|
|
|
|
|
|
|
|
""" Run the test.
|
|
|
|
|
|
|
|
|
|
The test needs to run self.rounds executing
|
|
|
|
|
self.operations number of operations each.
|
|
|
|
|
|
|
|
|
|
"""
|
|
|
|
|
# Init the test
|
|
|
|
|
a = 1
|
|
|
|
|
|
|
|
|
|
# Run test rounds
|
|
|
|
|
#
|
|
|
|
|
# NOTE: Use xrange() for all test loops unless you want to face
|
|
|
|
|
# a 20MB process !
|
|
|
|
|
#
|
|
|
|
|
for i in xrange(self.rounds):
|
|
|
|
|
|
|
|
|
|
# Repeat the operations per round to raise the run-time
|
|
|
|
|
# per operation significantly above the noise level of the
|
|
|
|
|
# for-loop overhead.
|
|
|
|
|
|
|
|
|
|
# Execute 20 operations (a += 1):
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
a += 1
|
|
|
|
|
|
|
|
|
|
def calibrate(self):
|
|
|
|
|
|
|
|
|
|
""" Calibrate the test.
|
|
|
|
|
|
|
|
|
|
This method should execute everything that is needed to
|
|
|
|
|
setup and run the test - except for the actual operations
|
|
|
|
|
that you intend to measure. pybench uses this method to
|
|
|
|
|
measure the test implementation overhead.
|
|
|
|
|
|
|
|
|
|
"""
|
|
|
|
|
# Init the test
|
|
|
|
|
a = 1
|
|
|
|
|
|
|
|
|
|
# Run test rounds (without actually doing any operation)
|
|
|
|
|
for i in xrange(self.rounds):
|
|
|
|
|
|
|
|
|
|
# Skip the actual execution of the operations, since we
|
|
|
|
|
# only want to measure the test's administration overhead.
|
|
|
|
|
pass
|
|
|
|
|
|
|
|
|
|
Registering a new test module
|
|
|
|
|
-----------------------------
|
|
|
|
|
|
|
|
|
|
To register a test module with pybench, the classes need to be
|
|
|
|
|
imported into the pybench.Setup module. pybench will then scan all the
|
|
|
|
|
symbols defined in that module for subclasses of pybench.Test and
|
|
|
|
|
automatically add them to the benchmark suite.
|
|
|
|
|
|
|
|
|
|
|
2006-05-26 19:33:20 -03:00
|
|
|
Breaking Comparability
|
|
|
|
|
----------------------
|
|
|
|
|
|
|
|
|
|
If a change is made to any individual test that means it is no
|
2006-05-26 19:39:27 -03:00
|
|
|
longer strictly comparable with previous runs, the '.version' class
|
2006-05-26 19:33:20 -03:00
|
|
|
variable should be updated. Therefafter, comparisons with previous
|
|
|
|
|
versions of the test will list as "n/a" to reflect the change.
|
|
|
|
|
|
2006-04-19 12:27:33 -03:00
|
|
|
Have fun,
|
|
|
|
|
--
|
|
|
|
|
Marc-Andre Lemburg
|
|
|
|
|
mal@lemburg.com
|
