Designing a Minimal Experiment

• Explain why an experiment's purpose is to distinguish a hypothesis from its stated alternative, not to gather data in general.
• Identify measurements in a proposed experiment that could not change the conclusion, and cut them.
• Design an experiment's stopping condition (how much data is "enough") before running it, rather than deciding after seeing results.
• Connect the size of a minimal experiment to the width of the confidence interval needed to tell a real effect from noise.
• Critique an over-built experiment by naming which of its measurements are actually load-bearing.

Once a hypothesis is stated precisely enough to be falsifiable, a natural next instinct is to gather as much data as possible about it — run every configuration, measure every variable, log everything, and sort it out afterward. This instinct is almost always a mistake, and correcting it is the point of this concept. An experiment is not a data-collection exercise; it is an act of discrimination between two specific possibilities — the hypothesis and its alternative. If a measurement would come out the same way regardless of which of the two is true, that measurement contributes nothing to the discrimination, no matter how interesting it looks. Designing a minimal experiment means asking, for every piece of data you are tempted to collect, "if this came out differently, would it change which of my two possibilities I believe?" — and refusing to collect it if the honest answer is no.

Richard Hamming pressed exactly this point in You and Your Research: people who do great work are typically working on one or two questions they have thought about hard enough to know precisely what would settle them, while people who accumulate data indiscriminately are usually avoiding the harder work of deciding, in advance, what would actually count as an answer. An experiment designed after that thinking is done tends to be strikingly small — often a single measurement, or a handful of trials — because most of what a sprawling, unfocused experiment would measure was never going to move the needle on the actual question. This is also where falsifiability, covered earlier in this discipline, cashes out as a design constraint rather than a philosophical remark: a hypothesis is only genuinely being tested if there exists some possible outcome that would make you abandon it, and a minimal experiment is built by asking specifically for the smallest observation that has this property.

The same discipline shows up constantly in software and systems work, where the temptation to over-measure is especially strong because instrumentation is cheap. Debugging a performance regression, for instance, invites logging every metric a profiler can produce — CPU time, memory allocation, cache misses, garbage-collection pauses, thread contention — across every code path. But if the hypothesis under test is "the regression is caused by repeated string concatenation in a hot loop," the discriminating experiment is much smaller: replace the concatenation with a pre-sized buffer, holding everything else fixed, and see whether the regression disappears. Every other metric collected alongside that one change is, for the purpose of this hypothesis, noise dressed up as rigor. Designing a minimal experiment is the skill of seeing that distinction before running anything, not after staring at a spreadsheet of results that don't actually settle the question.

The alternative defines the experiment

A hypothesis by itself does not fully determine what experiment to run — what determines it is the hypothesis together with the specific alternative you are trying to rule out. "This algorithm is O(n log n)" is testable against the alternative "this algorithm is O(n²)" with a fairly small experiment (a handful of input sizes spanning a couple of orders of magnitude, since the two curves diverge sharply and quickly). The same claim tested against the more specific alternative "this algorithm is O(n log n) but with a much larger constant factor than a competing implementation" requires a different, more carefully controlled experiment — the growth rate no longer discriminates, and now the constant factor, isolated from measurement noise, does. Before designing the experiment, the alternative has to be pinned down explicitly; "not doing more" is not itself an alternative that generates measurable predictions.

The cutting rule

For any measurement under consideration, ask: does the hypothesis predict a different value here than the alternative does? If both predict the same thing, or if neither makes any prediction about this quantity at all, the measurement is inert for the purpose of this experiment — it may be interesting for some other question, but it does not belong in this one. Applying this rule aggressively is what "minimal" means. It is not about running fewer trials for the sake of speed; it is about removing every measurement whose outcome is already known not to matter, however tempting it is to record it "just in case." A useful diagnostic: if you can predict, right now, before running anything, that a measurement will come out roughly the same either way, you already know it isn't part of the minimal experiment — cut it.

How much data is "enough": deciding before, not after

A minimal experiment is not just about which variables to measure — it is also about how many trials or how much data to collect before looking, and this has to be fixed in advance for the same reason falsifiability has to be fixed in advance: deciding afterward, once you can see how the data is trending, lets you unconsciously (or consciously) pick a stopping point that favors the conclusion you wanted. The right way to fix this in advance is to ask how large a difference between the hypothesis and the alternative would actually matter, and then determine how much data is needed to detect a difference of that size against the expected noise in the measurement. This is precisely the question a confidence interval answers on the statistical side (covered in full in Confidence Intervals, in the probability-and-statistics track): a confidence interval's width shrinks as sample size grows, and for two competing predictions to be distinguishable, the interval around your measured value needs to be narrow enough that it doesn't cover both the hypothesis's predicted value and the alternative's. Designing a minimal experiment and choosing a target confidence-interval width are the same act described from two different angles — the experimental design says "collect enough trials to tell these two predictions apart," and the interval's arithmetic says exactly how many trials that requires, given how much noise each individual measurement carries. Skipping this step and just "running it a few times" means the experiment's sample size was picked by convenience rather than by what the question actually demands, and a result that looks like a difference may simply be an interval wide enough to have swallowed both possibilities.

Recognizing an over-built experiment

An experiment that measures more than the minimal set is not automatically wrong — sometimes the extra measurements are cheap insurance against a hypothesis being underspecified, or against an alternative you hadn't thought of. But an over-built experiment carries real costs: it takes longer to run, it multiplies the number of ways an unrelated confound can creep in (a concern developed fully in Controls and Isolating Variables, the next concept in this track), and it produces a pile of results whose relevance to the actual question has to be re-litigated after the fact. A good check on a proposed experimental design is to go through every planned measurement and ask someone else to guess, without running anything, what result would falsify the hypothesis at that particular measurement. If they can't answer, or if their answer is "any result there is consistent with either possibility," that measurement is padding.

Example 1 — a caching hypothesis, trimmed to its minimal test

Setup. A hypothesis: "adding an in-memory LRU cache in front of this database lookup will reduce median request latency." The alternative: "the lookup is already fast enough, or the bottleneck is elsewhere, so the cache will make no measurable difference to median latency."

Over-built version. A first draft of the experiment logs CPU utilization, memory footprint, cache hit rate, 50th/90th/99th/99.9th percentile latency, database connection pool saturation, and garbage-collection pause times, across a full day of production traffic with the cache both enabled and disabled.

Trimming. Only one of these is actually discriminating between the hypothesis and its alternative: median request latency, with and without the cache, under otherwise identical load. Cache hit rate is informative about why the cache helps or doesn't, but the hypothesis as stated doesn't predict a specific hit rate — it predicts a latency change, so hit rate isn't required to settle the question (it becomes relevant only for a follow-up, more specific hypothesis). CPU, memory, GC pauses, and connection-pool saturation would only matter if the hypothesis were instead about resource consumption, not latency — the stated hypothesis makes no distinguishing prediction about any of them. Running a full day of production traffic is also unnecessary if the load can be replayed identically with and without the cache in a controlled setting; a full day adds volume, not discrimination, because time-of-day traffic variation is a confound, not a source of new information about this specific hypothesis.

Minimal experiment. Replay an identical, fixed request trace against the service twice — once with the cache enabled, once disabled — and measure only median latency in both runs, with enough repetitions of the replay to make the difference (if any) larger than the noise between runs. This is smaller, faster, and answers exactly the question asked.

Example 2 — sizing the experiment using confidence-interval logic

Setup. A hypothesis: "switching a sort routine from the current implementation to a new one reduces average execution time by at least 10%." The alternative: "the two implementations perform the same, within measurement noise."

Why one run isn't enough. A single timed run of each implementation could show a 10% difference purely from measurement noise (thread scheduling jitter, cache warmth, background system load) even if the true difference is zero, or could fail to show a real 10% difference if noise happens to work against it on that particular run. The experiment needs enough repeated trials that the confidence interval around the measured difference is narrower than 10% — narrow enough that "the true difference is zero" and "the true difference is at least 10%" cannot both plausibly fall inside it.

Working it through. Suppose preliminary timing shows individual runs vary by roughly ±3% from run to run (the noise). A single run's interval is far too wide to distinguish a 10% effect from no effect — the noise alone could produce a 3% swing in either direction. Averaging over many runs shrinks the interval's width roughly in proportion to the square root of the number of runs (the standard behavior of a mean's confidence interval); reaching a width comfortably under 10% — say, aiming for an interval of about ±2% around the mean — requires enough repetitions to shrink the ±3% single-run noise down to that target, which in practice means dozens of timed trials rather than one or two, alternating between the two implementations to average out any slow drift in system load.

Minimal design. Run both implementations interleaved (old, new, old, new, …) on the same machine under the same load, for the number of trials computed to bring the confidence interval on the mean difference below the 10% threshold that actually matters for the hypothesis — no more, since a narrower interval than needed only adds runtime without changing the conclusion, and no fewer, since a wider one risks a result that can't actually distinguish the two possibilities.

• "A bigger experiment is always more rigorous." Rigor comes from whether the experiment can actually distinguish the hypothesis from its alternative, not from how much data it produces. An experiment that measures ten irrelevant things alongside the one relevant one is not more rigorous than the minimal version — it is the same rigor, diluted with noise and false confidence from unrelated numbers that happened to look interesting.
• "I'll decide how much data is enough by seeing how the results look partway through." Choosing a stopping point after seeing preliminary trends is one of the most reliable ways to fool yourself, because it lets the data itself influence when you stop looking — typically right when it happens to support what you expected. The needed sample size should be computed from the size of the effect that would actually matter and the expected noise, before the first trial is run.
• "Minimal means quick and sloppy." Minimal refers to scope — how many distinct things are measured — not to care in execution. A minimal experiment can still require careful controls, repeated trials, and statistical rigor about sample size; it just refuses to measure things that were never going to change the answer.
• "If I don't know what the alternative is, I can just measure everything and figure out the hypothesis afterward." That is a different, legitimate activity (exploratory data analysis), but it is not hypothesis testing, and treating its output as if it had tested a specific claim is a common source of false confidence. A minimal experiment presupposes the hypothesis and its alternative are already both stated.

A minimal experiment is the smallest test that can actually distinguish a stated hypothesis from its stated alternative — every measurement that would come out the same regardless of which is true gets cut, no matter how easy it would be to collect. This starts from pinning down the alternative explicitly, since it is the pair of hypothesis and alternative, not the hypothesis alone, that determines what's worth measuring. It also requires fixing, in advance, how much data is needed to make a real difference detectable over noise — the same question a confidence interval's width answers directly: an experiment is appropriately sized when its result's confidence interval is narrow enough that it can't plausibly contain both the hypothesis's prediction and the alternative's. Over-built experiments aren't more rigorous; they add cost, invite confounds, and produce results whose relevance still has to be argued for after the fact, which is exactly the argument a well-designed minimal experiment settles before it is even run.