A word of warning when writing C# benchmarks

G.L Solaria - Sep 21 '22 - - Dev Community

I wrote a post on the free, open source tool BenchmarkDotNet but I wanted to mention a common pitfall that you might fall in to if you are not careful.

The warning is simple: when writing benchmarks, always use the results of your calculations.

Consider these benchmarks ...

[Benchmark(Baseline = true)]
public void Sqrt1()
{
    for (var ii = 0; ii < N; ++ii)
        Math.Sqrt(ii);
}

[Benchmark]
public double Sqrt2()
{
    double result = 0;
    for (var ii = 0; ii < N; ++ii)
        result = Math.Sqrt(ii);
    return result;
}
Enter fullscreen mode Exit fullscreen mode

If we look at the results of the above benchmarks ...

Method N Mean Error StdDev Ratio RatioSD Rank
Sqrt1 1000000 291.8 us 5.79 us 6.66 us 1.00 0.00 1
Sqrt2 1000000 4,006.9 us 55.00 us 51.45 us 13.64 0.29 2

How can Sqrt2 be almost 14 times slower than Sqrt1? The answer lies with the JIT compiler ...

SharpLab is another great tool that will show you what the JIT compiler will do initially. In the case of Sqrt1, it will compile out dead code because the result of the square root isn't used so it doesn't need to run the instructions to perform the square root.

So be careful with how you write your benchmarks otherwise you may not be measuring what you think you are measuring!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player