________________________________________________________________________ PYBENCH - A Python Benchmark Suite ________________________________________________________________________ Extendable suite of of low-level benchmarks for measuring the performance of the Python implementation (interpreter, compiler or VM). pybench is a collection of tests that provides a standardized way to measure the performance of Python implementations. It takes a very close look at different aspects of Python programs and let's you decide which factors are more important to you than others, rather than wrapping everything up in one number, like the other performance tests do (e.g. pystone which is included in the Python Standard Library). pybench has been used in the past by several Python developers to track down performance bottlenecks or to demonstrate the impact of optimizations and new features in Python. The command line interface for pybench is the file pybench.py. Run this script with option '--help' to get a listing of the possible options. Without options, pybench will simply execute the benchmark and then print out a report to stdout. Micro-Manual ------------ Run 'pybench.py -h' to see the help screen. Run 'pybench.py' to just let the benchmark suite do it's thing and 'pybench.py -f ' to have it store the results in a file too. This is the current output of pybench.py --help: Synopsis: pybench.py [option] files... Options and default settings: -n arg number of rounds (10) -f arg save benchmark to file arg () -c arg compare benchmark with the one in file arg () -s arg show benchmark in file arg, then exit () -S show statistics of benchmarks (0) -w arg set warp factor to arg (20) -d hide noise in compares (0) --no-gc disable garbage collection (0) --no-syscheck "disable" sys check interval (set to sys.maxint) (0) -t arg tests containing substring () -C arg number of calibration runs (20) -v generate verbose output -h show this help text --help show this help text --debug enable debugging --copyright show copyright --examples show examples of usage Version: 1.3 The normal operation is to run the suite and display the results. Use -f to save them for later reuse or comparisms. Examples: python1.5 pybench.py -w 100 -f p15 python1.4 pybench.py -w 100 -f p14 python pybench.py -s p15 -c p14 License ------- See LICENSE file. Sample output ------------- PYBENCH 1.3 Machine Details: Platform ID: Linux-2.6.8-24.19-default-x86_64-with-SuSE-9.2-x86-64 Executable: /home/lemburg/projects/Python/Installation/bin/python Python: 2.5a1.0 Compiler: GCC 3.3.4 (pre 3.3.5 20040809) Build: Apr 9 2006 01:50:57 (#trunk) Searching for tests... BuiltinFunctionCalls BuiltinMethodLookup CompareFloats CompareFloatsIntegers CompareIntegers CompareInternedStrings CompareLongs CompareStrings CompareUnicode ConcatStrings ConcatUnicode CreateInstances CreateStringsWithConcat CreateUnicodeWithConcat DictCreation DictWithFloatKeys DictWithIntegerKeys DictWithStringKeys ForLoops IfThenElse ListSlicing NestedForLoops NormalClassAttribute NormalInstanceAttribute PythonFunctionCalls PythonMethodCalls Recursion SecondImport SecondPackageImport SecondSubmoduleImport SimpleComplexArithmetic SimpleDictManipulation SimpleFloatArithmetic SimpleIntFloatArithmetic SimpleIntegerArithmetic SimpleListManipulation SimpleLongArithmetic SmallLists SmallTuples SpecialClassAttribute SpecialInstanceAttribute StringMappings StringPredicates StringSlicing TryExcept TryRaiseExcept TupleSlicing UnicodeMappings UnicodePredicates UnicodeProperties UnicodeSlicing Running 10 round(s) of the suite: ... Round 10 real abs overhead BuiltinFunctionCalls: 0.030r 0.030a 0.000o BuiltinMethodLookup: 0.059r 0.060a 0.001o CompareFloats: 0.050r 0.050a 0.000o CompareFloatsIntegers: 0.050r 0.050a 0.000o CompareIntegers: 0.070r 0.070a 0.000o CompareInternedStrings: 0.039r 0.040a 0.001o CompareLongs: 0.050r 0.050a 0.000o CompareStrings: 0.060r 0.060a 0.000o CompareUnicode: 0.060r 0.060a 0.000o ConcatStrings: 0.040r 0.040a 0.000o ConcatUnicode: 0.050r 0.050a 0.000o CreateInstances: 0.050r 0.050a 0.000o CreateStringsWithConcat: 0.029r 0.030a 0.001o CreateUnicodeWithConcat: 0.060r 0.060a 0.000o DictCreation: 0.040r 0.040a 0.000o DictWithFloatKeys: 0.089r 0.090a 0.000o DictWithIntegerKeys: 0.059r 0.060a 0.001o DictWithStringKeys: 0.070r 0.070a 0.001o ForLoops: 0.050r 0.050a 0.000o IfThenElse: 0.070r 0.070a 0.000o ListSlicing: 0.030r 0.030a 0.000o NestedForLoops: 0.030r 0.030a 0.000o NormalClassAttribute: 0.060r 0.060a 0.000o NormalInstanceAttribute: 0.060r 0.060a 0.000o PythonFunctionCalls: 0.060r 0.060a 0.000o PythonMethodCalls: 0.050r 0.050a 0.000o Recursion: 0.050r 0.050a 0.000o SecondImport: 0.030r 0.030a 0.000o SecondPackageImport: 0.030r 0.030a 0.000o SecondSubmoduleImport: 0.040r 0.040a 0.000o SimpleComplexArithmetic: 0.030r 0.030a 0.000o SimpleDictManipulation: 0.040r 0.040a 0.000o SimpleFloatArithmetic: 0.050r 0.050a 0.001o SimpleIntFloatArithmetic: 0.060r 0.060a 0.000o SimpleIntegerArithmetic: 0.060r 0.060a 0.000o SimpleListManipulation: 0.030r 0.030a 0.000o SimpleLongArithmetic: 0.030r 0.030a 0.000o SmallLists: 0.050r 0.050a 0.000o SmallTuples: 0.050r 0.050a 0.000o SpecialClassAttribute: 0.060r 0.060a 0.000o SpecialInstanceAttribute: 0.079r 0.080a 0.001o StringMappings: 0.060r 0.060a 0.000o StringPredicates: 0.049r 0.050a 0.001o StringSlicing: 0.039r 0.040a 0.000o TryExcept: 0.079r 0.080a 0.001o TryRaiseExcept: 0.059r 0.060a 0.001o TupleSlicing: 0.050r 0.050a 0.000o UnicodeMappings: 0.070r 0.070a 0.001o UnicodePredicates: 0.059r 0.060a 0.001o UnicodeProperties: 0.059r 0.060a 0.001o UnicodeSlicing: 0.050r 0.050a 0.000o ---------------------- Average round time: 2.937 seconds Tests: per run per oper. overhead ------------------------------------------------------------------------ BuiltinFunctionCalls: 29.85 ms 0.23 us 0.00 ms BuiltinMethodLookup: 66.85 ms 0.13 us 0.50 ms CompareFloats: 43.00 ms 0.10 us 0.00 ms CompareFloatsIntegers: 51.80 ms 0.12 us 0.00 ms CompareIntegers: 70.70 ms 0.08 us 0.50 ms CompareInternedStrings: 41.40 ms 0.08 us 0.50 ms CompareLongs: 47.90 ms 0.11 us 0.00 ms CompareStrings: 58.50 ms 0.12 us 0.50 ms CompareUnicode: 56.55 ms 0.15 us 0.50 ms ConcatStrings: 44.75 ms 0.30 us 0.00 ms ConcatUnicode: 54.55 ms 0.36 us 0.50 ms CreateInstances: 50.95 ms 1.21 us 0.00 ms CreateStringsWithConcat: 28.85 ms 0.14 us 0.50 ms CreateUnicodeWithConcat: 53.75 ms 0.27 us 0.00 ms DictCreation: 41.90 ms 0.28 us 0.00 ms DictWithFloatKeys: 88.50 ms 0.15 us 0.50 ms DictWithIntegerKeys: 62.55 ms 0.10 us 0.50 ms DictWithStringKeys: 60.50 ms 0.10 us 0.50 ms ForLoops: 46.90 ms 4.69 us 0.00 ms IfThenElse: 60.55 ms 0.09 us 0.00 ms ListSlicing: 29.90 ms 8.54 us 0.00 ms NestedForLoops: 33.95 ms 0.10 us 0.00 ms NormalClassAttribute: 62.75 ms 0.10 us 0.50 ms NormalInstanceAttribute: 61.80 ms 0.10 us 0.50 ms PythonFunctionCalls: 60.00 ms 0.36 us 0.00 ms PythonMethodCalls: 50.00 ms 0.67 us 0.00 ms Recursion: 46.85 ms 3.75 us 0.00 ms SecondImport: 35.00 ms 1.40 us 0.00 ms SecondPackageImport: 32.00 ms 1.28 us 0.00 ms SecondSubmoduleImport: 38.00 ms 1.52 us 0.00 ms SimpleComplexArithmetic: 26.85 ms 0.12 us 0.00 ms SimpleDictManipulation: 40.85 ms 0.14 us 0.00 ms SimpleFloatArithmetic: 48.70 ms 0.09 us 0.50 ms SimpleIntFloatArithmetic: 57.70 ms 0.09 us 0.00 ms SimpleIntegerArithmetic: 58.75 ms 0.09 us 0.50 ms SimpleListManipulation: 34.80 ms 0.13 us 0.00 ms SimpleLongArithmetic: 30.95 ms 0.19 us 0.50 ms SmallLists: 47.60 ms 0.19 us 0.00 ms SmallTuples: 48.80 ms 0.20 us 0.50 ms SpecialClassAttribute: 61.70 ms 0.10 us 0.00 ms SpecialInstanceAttribute: 76.70 ms 0.13 us 0.50 ms StringMappings: 58.70 ms 0.47 us 0.00 ms StringPredicates: 50.00 ms 0.18 us 1.00 ms StringSlicing: 39.65 ms 0.23 us 0.50 ms TryExcept: 84.45 ms 0.06 us 0.50 ms TryRaiseExcept: 61.75 ms 4.12 us 0.50 ms TupleSlicing: 48.95 ms 0.47 us 0.00 ms UnicodeMappings: 71.50 ms 3.97 us 0.50 ms UnicodePredicates: 52.75 ms 0.23 us 1.00 ms UnicodeProperties: 61.90 ms 0.31 us 1.00 ms UnicodeSlicing: 53.75 ms 0.31 us 0.50 ms ------------------------------------------------------------------------ Average round time: 2937.00 ms ________________________________________________________________________ Writing New Tests ________________________________________________________________________ pybench tests are simple modules defining one or more pybench.Test subclasses. Writing a test essentially boils down to providing two methods: .test() which runs .rounds number of .operations test operations each and .calibrate() which does the same except that it doesn't actually execute the operations. Here's an example: ------------------ from pybench import Test class IntegerCounting(Test): # Version number of the test as float (x.yy); this is important # for comparisons of benchmark runs - tests with unequal version # number will not get compared. version = 1.0 # The number of abstract operations done in each round of the # test. An operation is the basic unit of what you want to # measure. The benchmark will output the amount of run-time per # operation. Note that in order to raise the measured timings # significantly above noise level, it is often required to repeat # sets of operations more than once per test round. The measured # overhead per test round should be less than 1 second. operations = 20 # Number of rounds to execute per test run. This should be # adjusted to a figure that results in a test run-time of between # 20-50 seconds. rounds = 100000 def test(self): """ Run the test. The test needs to run self.rounds executing self.operations number of operations each. """ # Init the test a = 1 # Run test rounds # # NOTE: Use xrange() for all test loops unless you want to face # a 20MB process ! # for i in xrange(self.rounds): # Repeat the operations per round to raise the run-time # per operation significantly above the noise level of the # for-loop overhead. # Execute 20 operations (a += 1): a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 a += 1 def calibrate(self): """ Calibrate the test. This method should execute everything that is needed to setup and run the test - except for the actual operations that you intend to measure. pybench uses this method to measure the test implementation overhead. """ # Init the test a = 1 # Run test rounds (without actually doing any operation) for i in xrange(self.rounds): # Skip the actual execution of the operations, since we # only want to measure the test's administration overhead. pass Registering a new test module ----------------------------- To register a test module with pybench, the classes need to be imported into the pybench.Setup module. pybench will then scan all the symbols defined in that module for subclasses of pybench.Test and automatically add them to the benchmark suite. Breaking Comparability ---------------------- If a change is made to any individual test that means it is no longer strictly comparable with previous runs, the '.version' class variable should be updated. Therefafter, comparisons with previous versions of the test will list as "n/a" to reflect the change. Have fun, -- Marc-Andre Lemburg mal@lemburg.com