Hypothesis testing

The formula

The setup is always a pair of competing claims:

H₀: nothing changed        (null hypothesis — the boring explanation)
H₁: something changed      (alternative — the claim you'd like to make)

Reading: H₀ is the default. The data has to earn the right to reject it; you never get to assume the improvement.

The workhorse statistic for a measured mean against a target:

t = (x̄ − μ₀) / ( s / √n )        with n−1 degrees of freedom

Reading: how many standard errors separate your sample mean from the claimed value. The numerator is the effect you saw; the denominator is the wobble you'd expect in a sample mean built from n points with scatter s. When |t| beats the critical value from the t table, the difference is bigger than chance plausibly explains.

The p-value that comes out of it:

p = P( result at least this extreme │ H₀ is true )

Reading: if nothing changed, how often would raw luck hand you data this far out? Small p means either something changed or you just witnessed a rare accident. It is not the probability that H₀ is true — that misread causes more damage than any algebra error in this field ^[1]^[11].

The two ways to be wrong:

α = P( reject H₀ │ H₀ true )      Type I  — false alarm
β = P( keep  H₀ │ H₁ true )      Type II — missed detection
power = 1 − β

Reading: α is the false-alarm rate you chose (usually 0.05); β is the miss rate you probably never calculated ^[3]. Radar engineers will recognize this instantly: probability of false alarm, probability of detection. Same math, different vocabulary.

The sample size that a real test requires, for detecting a mean shift δ between two groups with scatter σ:

n per group ≈ 2 · ( (z_α/2 + z_β) · σ / δ )²

Reading: at α = 0.05 two-sided and 80% power, the constant 2·(1.96 + 0.84)² is about 15.7 — so detecting a one-sigma shift between two conditions takes roughly sixteen samples per group, and a half-sigma shift takes about sixty-three. Testing one measured mean against a known target drops the leading 2: about eight samples total for one sigma, thirty-two for half. Halve the effect, quadruple the data. One honesty note: the z-based shortcut runs slightly small at small n, because the real test uses the t distribution — the one-sample n = 8 actually delivers about 68% power, not 80. Add a sample or two, or iterate the calculation with t in place of z ^[3].

Where you meet it

The test stand, after a fix. You changed the isolator mounts and the vibration response looks lower — 4.6 g RMS average over eight runs against 5.0 before, run scatter about 0.5 g. That's t = 1.6, p ≈ 0.13. The review board should not accept "the fix worked" from that data, and a two-sample t-test is how you know before they do.
Acceptance sampling on incoming hardware. Every sampling plan in the MIL-STD-105 lineage is a hypothesis test wearing shop clothes: producer's risk is α (good lot rejected), consumer's risk is β (bad lot accepted), and the operating characteristic curve is the power calculation drawn as a picture ^[3].
Reliability demonstration. "Demonstrate 2,000-hour MTBF at 90% confidence" is a hypothesis test on failure data — the test duration and allowable failure count fall straight out of the α and β the customer signed up for.
The design review, defending a claim. Anyone can put two bar charts on a slide. The question a sharp reviewer asks — "how many units, and what's the scatter?" — is a request for the t-statistic, whether or not anyone says the word.

How it works

The logic is proof by statistical contradiction. Assume the boring explanation, compute how surprising your data would be under it, and if the surprise is extreme, abandon the assumption. The machinery — t, z, chi-squared, F — differs only in what's being compared and what's known; the skeleton never changes: statistic, reference distribution, threshold, decision.

The threshold is a knob, not a law. Setting α = 0.05 means you accept one false alarm per twenty tests of true nulls. That's the actual operational meaning, and it compounds: run twenty comparisons on one dataset at 0.05 each and there's a 64% chance at least one comes up "significant" by pure luck. That's the mechanism behind p-hacking — keep slicing the data until something clears the bar, then report only the slice.

What people get wrong, in rough order of expense:

Failing to reject ≠ proving the null. The isolator example above has p = 0.13 — but with eight runs per condition, the power to detect a true 0.4 g improvement is only about 32%. The test was two-to-one likely to miss a real effect. "Not significant" from an underpowered test means "didn't look hard enough," not "no difference." Compute the power, or the sample size formula above, before you run — not after.
Statistical significance ≠ engineering significance. With n = 10,000 telemetry frames, a 0.01 dB difference will be wildly significant and completely irrelevant. The p-value says "probably real"; only the effect size says "worth caring about." Report both.
Ignoring independence. The t-test assumes independent samples. A thousand points from one continuous run of a test stand are not a thousand independent samples — drift and autocorrelation can make the effective n a tenth of the nominal count, and the standard error a factor of three optimistic. Replicate runs, not just samples within a run.
Assuming normality the data doesn't have. The t-test also assumes the underlying population is roughly normal. The central limit theorem makes it forgiving at moderate n, but below about ten samples — exactly the sizes in the examples here — skewed data can make the nominal 0.05 false-alarm rate meaningfully wrong, and the quantities engineers test are often skewed: failure times, peak levels, anything bounded at zero. Check a normal probability plot first; if it bends, transform (log fixes most lifetime and level data) or switch to a nonparametric test (Wilcoxon signed-rank, Mann–Whitney) that doesn't care about the shape ^[3].
One-sided when you promised two-sided. Deciding the test direction after seeing the data quietly doubles your false-alarm rate. Pick the tails before the test — that's the whole discipline Neyman and Pearson were selling ^[9].
Worshipping 0.05. p = 0.049 and p = 0.051 are the same evidence. The American Statistical Association said as much in a 2016 statement: a p-value alone is not a good measure of evidence, and "significant" is not a synonym for "true" ^[1]^[11].

Small-sample reality: with n = 3 the critical t at 95% two-sided is 4.30, against 1.96 for large n. Three data points must show an enormous effect before the math will back you. That's not the test being difficult; that's the test being honest about how little three points know.

History

The first recognizable significance test predates the discipline by two centuries. In 1710, John Arbuthnot — physician to Queen Anne and a satirist on the side — noticed that London christening records showed more boys than girls in every single year from 1629 to 1710, all 82 of them. If sex were a coin flip, the chance of that streak is (1/2)⁸², about 2·10⁻²⁵. He took it as proof of divine providence; statisticians take it as the first formal significance argument on record ^[1]^[4].

The modern toolkit assembled between 1900 and 1933. Karl Pearson's chi-squared test of 1900 introduced the capital-P probability that a discrepancy could arise from random sampling ^[1]^[12]. In 1908, a Guinness brewer named William Sealy Gosset published "The Probable Error of a Mean" in Biometrika — the small-sample t-test, built because brewery experiments on barley couldn't afford large n. Guinness, burned once by an employee publishing competitive details, required pen names, so the paper appeared under "Student," which is why the distribution carries a pseudonym instead of a name ^[5]^[6]^[7].

Ronald Fisher, working through crop-trial data at Rothamsted Experimental Station from 1919, wove these pieces into a working method. His 1925 Statistical Methods for Research Workers put significance testing in every lab, and his suggestion of one-in-twenty as a convenient cutoff is where 0.05 comes from — a convenience he himself later said should bend to circumstances, advice the world ignored ^[2]^[8].

The framework engineers actually use arrived in 1933, when Jerzy Neyman and Egon Pearson (Karl's son) published "On the Problem of the Most Efficient Tests of Statistical Hypotheses." They added what Fisher's approach lacked: an explicit alternative hypothesis, the two error types, power, and a proof — the Neyman–Pearson lemma — that the likelihood ratio test is the best detector available ^[9]^[10]. Fisher hated it and said so for decades. Modern practice is an uneasy fusion of the two camps, which is part of why the p-value gets misread: it's a Fisher quantity living inside a Neyman–Pearson decision procedure. The 2016 ASA statement was the profession's formal attempt to clean up the resulting confusion ^[11].

Related tools

/tools/bearing-life-l10 — L10 life is a statistical claim about a population, the same kind of assertion a hypothesis test defends
/tools/rms-peak — the sample statistics you compute before any test statistic
/tools/thermal-noise-floor — the physical source of the randomness the t-test is built to see through
/tools/snr-enob — signal-versus-noise as a ratio; the t-statistic is the same idea applied to a measured mean

Sources

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →