t-tests

The formula

One-sample — is my measured mean different from the spec value μ₀?

t = (x̄ − μ₀) / (s / √n),   df = n − 1

Reading: the observed gap between sample mean and spec, measured in units of the standard error of the mean. Big |t| means the gap is large compared to what sampling noise alone would produce.

Two-sample, pooled (classic Student, assumes equal variances):

t = (x̄₁ − x̄₂) / ( s_p · √(1/n₁ + 1/n₂) ),   df = n₁ + n₂ − 2
s_p = √( ((n₁−1)·s₁² + (n₂−1)·s₂²) / (n₁ + n₂ − 2) )

Reading: the gap between two group means over a pooled estimate of the shared scatter. s_p is just the two sample variances averaged, weighted by their degrees of freedom.

Two-sample, Welch (doesn't assume equal variances — the one you should default to):

t = (x̄₁ − x̄₂) / √( s₁²/n₁ + s₂²/n₂ )
df ≈ (s₁²/n₁ + s₂²/n₂)² / ( (s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1) )

Reading: same gap, but each group carries its own variance, and the Welch–Satterthwaite formula charges you degrees of freedom for the mismatch.

Paired — same units measured before and after: take the per-unit differences d_i = x_i,after − x_i,before and run the one-sample test on them against μ₀ = 0. Pairing subtracts out unit-to-unit variation, which is usually the biggest noise source on real hardware.

In every case you compare |t| against a critical value from the t-distribution at your df, or equivalently read off a p-value. At 95% two-sided confidence the critical value is 2.776 at df = 4, 2.262 at df = 9, 2.093 at df = 19, and 1.960 in the large-sample limit — small samples pay a real penalty for estimating σ from the data.

Where you meet it

Acceptance against spec. Five thrust measurements off the stand average 10.20 klbf against a 10.00 spec, with s = 0.30. Then t = 0.20 / (0.30/√5) = 1.49 on 4 degrees of freedom — well inside 2.776, p ≈ 0.21. The data can't distinguish that motor from nominal. Whether that's good news depends on which side of the review board table you sit on.
Vendor A versus vendor B. Eight connectors from each vendor, insertion-loss means 52.1 and 50.8 (s = 1.2 and 2.3). Welch gives t = 1.42 on df ≈ 10.5, against a critical 2.21. No demonstrated difference — and note the pooled test gives the identical t = 1.42 here because the groups are the same size; what changes is the df you're charged.
Before/after on the same units. Ten flight boxes measured for gain, sent through a rework, measured again. Paired t on the ten differences. Running an unpaired test on this data instead throws away the pairing and buries a real rework effect under box-to-box spread.
Cal-lab drift checks. A reference standard measured monthly against its certified value — the one-sample t is the formal version of "is this thing still reading true, or am I looking at noise?"

How it works

The whole point of the t-test is that you don't know the true standard deviation σ — you estimated s from the same handful of data points, and s from a small sample is itself noisy. Dividing by a noisy s makes extreme t values more common than the normal distribution predicts, so the t-distribution has fatter tails, converging to the normal as n grows. Ignore this and use the familiar ±1.96 cutoff on 5 data points and your actual false-alarm rate is about 12%, not the 5% you think you're running.

Gotchas, in the order they'll bite you:

Failing to reject is not proving equal. With 5 units per group, a true difference of one full standard deviation gets flagged only about 29% of the time (power ≈ 0.29). "The t-test showed no difference" on a thin sample mostly shows you didn't collect enough data. If the program needs a demonstration of equivalence, that's a different test (TOST) with its own sample-size math.
Statistical significance is not engineering significance. With n large enough, a 0.01 dB difference between vendors will come back p < 0.05. The test answers "is the difference real?", never "does it matter?" Put the engineering threshold in writing before you look at the p-value.
Default to Welch. The pooled test's equal-variance assumption fails quietly, and it fails worst when the smaller sample has the bigger variance. Welch costs essentially nothing when variances really are equal, so the two-step ritual of variance-testing first and choosing after is both unnecessary and mildly harmful. Modern practice: just run Welch.
Independence is the assumption that actually kills you. Normality matters less than people fear — means average out toward normal, and the test is tolerant of moderately non-normal data. But 30 readings logged over 10 seconds from a drifting instrument are not 30 independent samples; the effective n is far smaller, s/√n is a lie, and the t value is inflated. Autocorrelated data is the single most common way engineers get a false positive out of this test.
Outliers own the result. Both x̄ and s are wide open to a single bad reading. One dropped bit in a datalog can manufacture or erase a significant result. Plot the raw data before you quote the p-value — every time.

History

The t-test was invented to solve a brewery problem. William Sealy Gosset, an Oxford-trained chemist, joined the Guinness brewery in Dublin in 1899 and spent his career running small experiments — barley varieties, hop batches, yeast counts — where collecting thirty samples was economically absurd and the large-sample theory of the day gave misleading answers ^[2]^[3]. Guinness sent him to Karl Pearson's laboratory at University College London for the 1906–07 year to work out the mathematics of small samples ^[2]^[3].

The result was "The Probable Error of a Mean," published in Biometrika in March 1908 — under the pseudonym "Student," because after an earlier employee had published trade information, Guinness barred its researchers from publishing under their own names ^[1]^[2]. Gosset originally called his statistic z; it was Ronald Fisher who, starting from a 1912 letter pointing out the statistic should be scaled by degrees of freedom rather than sample size, reworked the theory and gave the modern test its form and its letter t ^[1]^[2]. Fisher and Gosset traded over 150 letters across two decades ^[2]. The name "Student's t-test" stuck, so the most-used hypothesis test in engineering is credited to a man whose employer wouldn't let him sign it.

The unequal-variance version came four decades later: Bernard Lewis Welch of the University of Leeds published the generalization in Biometrika in 1947 ^[4]^[5], with the approximate degrees-of-freedom formula now known as the Welch–Satterthwaite equation ^[4]^[6] — the same machinery, incidentally, that combines uncertainty terms in every modern cal-lab uncertainty budget.

Related tools

/tools/strain-gauge-bridge — the bench measurement whose repeat readings end up as the x_i in a one-sample t
/tools/adc-resolution — quantization noise is one floor under the s your t-test divides by
/tools/snr-enob — instrument noise budget; decides how much of your sample scatter is the DUT versus the digitizer
/tools/bearing-life-l10 — the reliability-side cousin: comparing vendor life data needs Weibull methods, not a t-test on means

Sources

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →