Measuring code performance with Benchmark.NET

Today we are going to look at how we can use Benchmark.Net to measure code performance. With benchmarks we can create baselines to verify changes made work as expected and don’t cause performance regressions. Software development is often a science experiment. We create an hypothesis on the best way to solve a problem, write some code, and then run an experiment to test that hypothesis. Did it run correctly, fast (enough), and is the code still maintainable? We run this little feedback loop many, many times as we write each line of code.

Benchmarking isn’t needed for every project, but if you are developing a NuGet package, or DLL that will do processing, it makes a lot of sense. We are going to use it to tackle an age old problem, string appending. The classic example is that appending a string with the + operator is relatively expensive, for example:

string myString = "string1" + "string2" + "string3" + "string4" + "string5";
Console.WriteLine(myString);

However, many of us are aware that StringBuilder is recommended as an alternative and much faster, especially if you have many strings (which we classify as more than three):

StringBuilder sb = new StringBuilder();
sb.Append("string1");
sb.Append("string2");
sb.Append("string3");
sb.Append("string4");
sb.Append("string5");
Console.WriteLine(sb.ToString());

Using Benchmark.NET

How do we bench mark this? Let’s create a project and walk through the basic steps. First we need to create a console project:

Next we add the NuGet package Benchmark.NET.

Then, in the program.cs file, we add two BenchMarkDotNet using commands (lines 1 and 2), and change the Program accessibility to “public” (line 8).

Now we can create some tests. Each test needs the “[Benchmark]” attribute. We create three examples to append the strings with different methods, using the + operator, the “Concat” function from Linq, and using StringBuilder.

Finally, in the Main entrypoint to our console app, we add the BenchmarkRunning Run command.

We browse to the command line and run the benchmarking project:

dotnet run -p BenchmarkTesting.app.csproj -c Release

This takes about a few minutes to run. Behind the scenes the benchmark tool is creating a number of threads/jobs and running the tests multiple times. For consistent results, it’s recommended to minimize open applications and not do anything else while this is running. The output is shown below, with a LOT of information.

Let’s look at the table embedded in the last image closer – these are our experiment results. It’s clear from this that StringBuilder is much more efficient than the + operator. The Linq Concat function is relatively efficient too, but can’t compete with StringBuilder. While some teams download these and make visuals, it’s clear to us who the winners and losers are here.

• Error is half of the 99.9% confidence interval
• The unit “us” is short for “μs”, or microseconds. A microsecond is equal to ¹⁄_1,000 of a millisecond.

Wrap-up

Use stringbuilder when appending strings! Benchmarking is a useful tool for measuring performance. It’s worth noting that with the effort required, this won’t work for every project, but for testing NuGet packages or DLL’s, this is a great way to test. There isn’t currently a good DevOps story for benchmarking, partly because there is variability in results from run to run, but watch this space!

References:

• NuGet package: https://www.nuget.org/packages/BenchmarkDotNet/
• Official documentation: https://benchmarkdotnet.org/articles/overview.html
• Featured image credit