Benchmarking with BenchmarkDotNet and Allocation Profiling

"This version is faster" and "this version allocates less" are claims about numbers, and a number taken with a stopwatch around one call is rarely the number you think it is. BenchmarkDotNet is the standard tool for measuring .NET code: it handles warm-up, JIT tiers, process isolation and statistics, and with [MemoryDiagnoser] it reports allocations per operation, which is the number that usually explains a slow service. This concept shows why a stopwatch misleads, how to write a benchmark with a baseline and parameters, how to read the report, which traps make a benchmark measure nothing, how pooling looks in the numbers, and how to watch allocation in a running process. Every number below was measured on one machine (an Apple M4 Max, macOS, arm64, 14 cores, BenchmarkDotNet 0.15.8). Treat them as an illustration of the method, not as data about your hardware.

• Proving, before a pull request, that a rewrite allocates less and is not slower.
• Choosing between two implementations (+= versus StringBuilder, new versus ArrayPool) with evidence instead of folklore.
• Reading Gen0 and Allocated to find out why a hot path triggers many collections.
• Checking that a runtime upgrade did not regress a critical method.
• Watching the allocation rate of a running service without attaching a debugger.

Why a stopwatch misleads

A loop of Stopwatch.StartNew() around a call looks fine until you call it more than once. Building a semicolon-separated string from 100 integers with +=, timed on five consecutive calls in a Release build:

csharp
for (int i = 0; i < 5; i++)
{
    long before = GC.GetAllocatedBytesForCurrentThread();
    var sw = Stopwatch.StartNew();
    var s = Csv.Plus(values);
    sw.Stop();
    Console.WriteLine($"{sw.Elapsed.TotalMicroseconds:F1} us, {GC.GetAllocatedBytesForCurrentThread() - before} B");
}

The first call was 25 to 60 times slower than the steady state, because it includes JIT compilation and first-use costs. The steady state needs several calls to appear, tiered compilation can still change the code afterwards, and a single number has no spread. BenchmarkDotNet exists to remove exactly these problems: it runs the method in a separate process, calibrates how many operations to run per measurement, discards warm-up, repeats, and reports the mean with an error interval. Note that GC.GetAllocatedBytesForCurrentThread already gave the stable number here (30,944 bytes) after the first call. Bytes are far more repeatable than time, which is why the lab for this concept asserts on them.

A first benchmark

csharp
[MemoryDiagnoser]
public class ConcatBench
{
    [Params(10, 1000)] public int N;
    private int[] _values = [];

    [GlobalSetup] public void Setup() => _values = Enumerable.Range(0, N).ToArray();

    [Benchmark(Baseline = true)] public string Plus() => Csv.Plus(_values);
    [Benchmark] public string Builder() => Csv.Builder(_values);
}

Run it from a console project that references the BenchmarkDotNet package, in Release, outside the debugger:

csharp
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

plaintext
dotnet run -c Release -- --filter "*ConcatBench*" --job short

--job short runs fewer iterations (three here), which is why the error column below is wide. Use the default job for a conclusion you will defend.

Reading the report

• Baseline and Ratio. [Benchmark(Baseline = true)] marks the reference method; Ratio is each method's mean divided by the baseline's mean (the report legend says "[Current]/[Baseline]"), so 0.02 means about fifty times faster. A ratio is only meaningful within one report.
• [Params] runs every benchmark once per value and prints one block per value, separated by an empty row. The table above shows why: at N = 10 the builder is 33 percent faster, at N = 1000 it is almost fifty times faster and allocates 230 times fewer bytes, because += copies the whole string at each step and the cost grows with the square of the input. One input size would have hidden the story.
• Gen0, Gen1, Gen2 are collections per 1000 operations: 459 gen0 collections per 1000 calls of Plus(1000). This is the cost the mean does not show, because it falls on the whole process.
• Allocated is managed memory per operation, inclusive of what the method calls; native memory is not counted.
• Error and StdDev tell you how much to trust Mean. If the error is a large fraction of the mean (the N = 10 row above, from a short run), do not conclude anything from that row.

Traps that make a benchmark measure nothing

Four methods that look like benchmarks, in one class with [MemoryDiagnoser]:

csharp
[Benchmark] public void Dead()     { var a = new byte[1000]; }
[Benchmark] public int  Kept()     { var a = new byte[1000]; return a.Length + _x; }
[Benchmark] public byte[] Returned() { var a = new byte[1000]; return a; }
[Benchmark] public double Constant() => Math.Sqrt(2.0);

BenchmarkDotNet printed a ZeroMeasurement warning for the first, second and fourth: "the method duration is indistinguishable from the empty method duration". The JIT removed the array that nothing used, even in Kept, where only its length was read (the escape analysis described in Value Types, Reference Types and Boxing), and folded Math.Sqrt(2.0) into a constant. Only the version that returns the array measured an allocation. The rules that follow:

• Return the result (or hand it to a Consumer/Blackhole) so the work cannot be removed. This is dead code elimination.
• Read inputs from fields set in [GlobalSetup], never constants, so the JIT cannot fold the computation.
• Keep setup out of the method. Building the input inside the benchmark measures the setup too; [GlobalSetup] runs once and [IterationSetup] is a last resort, because it distorts short benchmarks.
• Release only. Running the same project in Debug, BenchmarkDotNet stops before measuring: "Assembly Demo which defines benchmarks is non-optimized". Also run outside the debugger and, ideally, with other heavy programs closed.
• Watch for tiering and warm-up. The default job runs warm-up iterations until the method is stable; if you lower it, check that Mean is not still falling.

Comparing .NET 10 and .NET 11

BenchmarkDotNet can run the same benchmark on several runtimes through a job per runtime. On 0.15.8, the stable version when this was written, that is a problem for .NET 11: its CoreRuntime has Core10_0 and no Core11_0, the command line --runtimes net11.0 is rejected ("The provided runtime "net11.0" is invalid", the list ends at net10_0), and a project built for net11.0 fails with NotImplementedException: GetRuntimeVersion not implemented for NotRecognized in the default toolchain. What worked here was building the project for both target frameworks and running each build in the process with --inProcess, which measures on the host runtime without the SDK check:

plaintext
dotnet run -c Release -f net10.0 -- --filter "*VersionBench*" --inProcess
dotnet run -c Release -f net11.0 -- --filter "*VersionBench*" --inProcess

Differences of 1 to 4 percent with errors up to 2 percent: no conclusion beyond "no regression here". --inProcess gives up the process isolation that protects a benchmark from the host's own state, so it is a workaround, not a default; check the release notes of BenchmarkDotNet for first-class .NET 11 support before relying on this.

Pooling as a measured fix

ArrayPool<T>.Shared and ObjectPool<T> (package Microsoft.Extensions.ObjectPool) replace allocation with reuse. Whether they pay off is a question for the report:

The array above 85,000 bytes lives on the large object heap, so every call feeds gen2 collections (the 31 in the Gen2 column); the pool removes the allocation altogether. The small builder case is a modest gain, and the pooled version still allocates 40 bytes for the result string. Pooling has a configuration cost you can measure as well. The default policy of the StringBuilder pool discards builders that grew past 4,096 characters, so with a result of 3,890 characters it pays for the pool plus a new builder every call:

At 100 values the pool allocates only the result (602 B for a 290 character string). At 1000 values the pooled version allocated more than a plain builder, because the builder never survived the pool's size limit (the policy's InitialCapacity is 100 and MaximumRetainedCapacity is 4,096, read from StringBuilderPooledObjectPolicy). A pool is a trade, not a free win: measure with your real sizes. The same holds for ArrayPool (clear sensitive data on Return with clearArray: true, and never touch a buffer after returning it).

Allocation in a running process

When you cannot attach a benchmark, dotnet-counters reads the runtime's counters through EventPipe. Installed with dotnet tool install dotnet-counters --tool-path <dir> (version 10.0.750501 here) and pointed at a process that called the += method in a loop for 25 seconds:

plaintext
dotnet-counters collect -p <pid> --counters System.Runtime \
    --format csv -o counters --refresh-interval 1 --duration 00:00:08

The CSV had one row per counter per second, using the OpenTelemetry style names of recent runtimes. The ones that answer an allocation question:

That is a gen0-only storm: everything dies young, nothing reaches the LOH, and the process spends its time allocating and collecting. It is the shape of the += loop with 1000 values from the benchmark above. Counters tell you that a process allocates too much and in which generation; to find which methods and types are responsible, use an allocation profile or a dump with the tools in Diagnosing Production with dotnet-trace, dump and gcdump, then confirm the fix with a benchmark.

• Stopwatch versus BenchmarkDotNet. A stopwatch is enough to see an order of magnitude (a 50x difference is not noise); BenchmarkDotNet is for decisions, at the price of minutes per run and a separate project.
• Short job versus default job. --job short makes a fast feedback loop but produces wide error bars, as the N = 10 row shows; use the default job for numbers that go into a pull request.
• Time versus allocations. Bytes per operation are stable across machines and runs, time is not. For regression tests, assert on allocation in ranges (as the lab does with GC.GetAllocatedBytesForCurrentThread) and keep time in benchmarks you read.

  csharp
  var before = GC.GetAllocatedBytesForCurrentThread();
  Work();
  Assert.InRange(GC.GetAllocatedBytesForCurrentThread() - before, 0, 1_000);

• Microbenchmark versus production. A benchmark on one machine, one input size and a warm cache says little about contention, a loaded GC heap or a different CPU. Use it to compare two versions, then confirm in production with counters.
• Pooling versus plain allocation. A pool removes GC pressure for big or hot buffers, and adds lifetime rules (return it, do not touch it after) and size policies. For small short-lived objects, the generational GC is often cheaper than the discipline.