Statistical power and sample size

The formula

Power is defined against the two ways a test goes wrong. Reject a true null hypothesis and you've made a Type I error, with probability α; miss a real effect and you've made a Type II error, with probability β ^[1]^[6]:

power = 1 − β = P( reject H₀ | the true difference is δ )

Reading: power is the detection probability of the test, evaluated at a specific, stated effect size δ. There is no such thing as "the" power of a test — only its power against a particular difference.

Turn that around and it becomes a sample-size equation. To detect a shift δ in a mean, with noise σ, two-sided significance α, and power 1 − β ^[2]:

n = (z₁₋α/₂ + z₁₋β)² · (σ/δ)² (one sample vs. a known reference) n = 2 · (z₁₋α/₂ + z₁₋β)² · (σ/δ)² (per group, comparing two test groups)

Reading: the sample size is a squared sum of two normal quantiles — one bought by your false-alarm tolerance, one by your miss tolerance — scaled by the noise-to-signal ratio squared. The workhorse values: z₀.₉₇₅ = 1.960, z₀.₈₀ = 0.842, z₀.₉₀ = 1.282, z₀.₉₅ = 1.645.

For the standard recipe (α = 0.05 two-sided, 80% power), (1.960 + 0.842)² = 7.85, and the two-group version collapses to Lehr's rule of thumb ^[1]:

n ≈ 16 / Δ² per group, where Δ = δ/σ

Reading: sixteen over the effect size squared. Detect a one-sigma shift: 16 per group. Half a sigma: 64 per group. A quarter sigma: 256. The inverse-square is the whole story of test cost.

The attribute-test cousin, for demonstrating reliability R at confidence C with zero allowed failures (the success-run formula) ^[10]^[11]:

n = ln(1 − C) / ln(R)

Reading: 22 units for 90% reliability at 90% confidence, 59 for 95/95, 299 for 99% reliability at 95% confidence. All must pass; one failure and the demonstration is over.

Where you meet it

The qual review board. The program asks how many units prove 95% reliability at 95% confidence. The success-run formula says 59, zero failures allowed, and no amount of viewgraph engineering moves that number. When the budget holds 8 units, the honest options are lowering the demonstrated reliability, raising the risk accepted, or bringing prior data — not testing 8 and hoping.
The test stand. A new igniter lot needs to match the old one's mean chamber pressure. If unit-to-unit scatter is σ and a shift of 0.5σ matters, the two-group formula demands 63 firings per lot for 80% power. The ten-and-ten plan someone sketched in the tag-up has 20% power against that same shift — it will miss the problem four times out of five.
The bench. Comparing two torque tools, two adhesives, two connector platings: before collecting data, the only question that matters is what difference is worth detecting. Power analysis converts that engineering judgment into a sample count; skipping it converts the comparison into a ritual.
The proposal. Test programs get costed by unit count. Power analysis is the auditable line between the statistics section and the price volume — and the defense when a customer asks why the plan says 22 units instead of 5.

How it works

Five quantities are chained together: α, β, δ, σ, and n. Fix any four and the fifth is determined. Every sample-size calculation, no matter how fancy the software, is a rearrangement of that chain — and the engineering content lives entirely in two of the five: δ, the smallest difference that matters to the hardware, and σ, the noise you honestly expect. The z-values are table lookups; the judgment is not.

A worked case. Thrust scatter on a motor family runs σ = 40 lbf, and a shift of δ = 25 lbf from nominal would matter. Two-sided α = 0.05, 80% power, one test group against a known nominal:

n = (1.960 + 0.842)² · (40/25)² = 7.85 · 2.56 = 20.1 → 21 firings

Want 90% power instead? (1.960 + 1.282)² · 2.56 = 26.9 → 27. Want to resolve δ = 12.5 lbf? Four times the units — 81. The one-sided version of the same machinery, sizing a test to catch a one-sigma degradation with α = 0.05 and 90% power, gives (1.645 + 1.282)² = 8.57 → 9 units, which is NIST's own worked example ^[2].

One caveat on all of these counts: the z formulas assume a normally distributed response and a σ known in advance. When σ will itself be estimated from the test data — the usual case — the real test is a t-test and the z answer runs small: the 21-firing plan above actually delivers about 78% power, and exact sizing against the noncentral t distribution, the iteration NIST's procedure walks through ^[2], gives 23. At sample sizes in the tens, padding the z answer by about two units closes the gap.

The behavior worth internalizing:

Cost goes as the inverse square of the effect. Halving the detectable difference quadruples the test. This is why "let's also check for small shifts while we're at it" is a sentence that doubles a program.
Power is bought before the test, not computed after. Running the data through a power calculator after a non-significant result — "observed power" — is circular: it is a deterministic function of the p-value you already have and adds no information ^[1]. If the test came back non-significant, the useful artifact is the confidence interval on the difference, which shows what the test could and couldn't rule out.
Failing to find a difference is not finding no difference. An underpowered test that comes back non-significant proves nothing except that the test was small. If the program needs to demonstrate equivalence — new vendor equals old vendor — that is an equivalence test with its own sample-size arithmetic, and the burden of proof reverses.
The σ guess is the soft spot. Sample size is proportional to σ², and σ is usually estimated from a handful of prior units or a similar program. Underestimate scatter by 30% and the "80% power" test is really a 60% test. Sizing from a pilot run, or padding n against the upper confidence bound on σ, is cheap insurance.
One-sided vs. two-sided is a real decision. If only degradation matters, z₁₋α replaces z₁₋α/₂ and the test gets smaller — but the choice has to be defensible before the data exists, not selected afterward because it flatters the result.
n means independent articles. Ten firings of one unit measure that unit, not the population. Repeated measurements shrink measurement noise; only more serial numbers shrink unit-to-unit uncertainty. Conflating the two is the most common way a sample-size claim dies in review.

History

The framework is a product of the Neyman–Pearson collaboration. Egon Pearson, at University College London, wanted a general principle behind the small-sample tests Gosset had built for Guinness; he and Jerzy Neyman agreed to work the problem together in the mid-1920s and carried it on by correspondence between London, Paris, and Warsaw ^[4]^[5]. Their answer reframed testing entirely: a test is a decision between a null and an alternative, it can fail in exactly two ways — the false alarm and the miss, the two error types that now anchor every test plan ^[3]^[6] — and among all tests holding the false-alarm rate at α, the likelihood-ratio test maximizes the probability of catching the alternative. That optimality result is the Neyman–Pearson lemma, published in their 1933 paper "On the problem of the most efficient tests of statistical hypotheses" in the Philosophical Transactions of the Royal Society ^[3]^[4]. Power was born there, as the quantity a good test maximizes.

Making engineers and scientists actually use it took another generation, and the pusher was a psychologist. Jacob Cohen (1923–1998) of New York University pulled every 1960–61 article from the Journal of Abnormal and Social Psychology and computed the power its studies had against small, medium, and large effects; the 1962 review found researchers policed α carefully while running studies with roughly coin-flip power against medium effects — tests built to miss ^[7]^[8]^[9]. His fix was practical: tables, the standardized effect size d = δ/σ with its small/medium/large landmarks of 0.2, 0.5, and 0.8, and the 80%-power convention, codified in Statistical Power Analysis for the Behavioral Sciences (second edition 1988) ^[1]^[7]. A follow-up survey of the same journal's 1984 volume, published a quarter century after Cohen's, found average power essentially unchanged ^[12] — a standing reminder that the math has been settled since 1933 and the discipline is the hard part.

Related tools

/tools/bearing-life-l10 — L10 life is a reliability quantile; demonstrating it on real units runs through the same confidence-versus-sample-count arithmetic as the success-run formula.
/tools/snr-enob — detection probability against noise is the same problem in signal clothing; a statistician's power curve is an RF engineer's probability of detection.

Sources

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →