Multiple linear regression

The formula

The model:

y = β₀ + β₁·x₁ + β₂·x₂ + … + β_p·x_p + ε

Read: the response is a baseline plus a weighted sum of the predictors, plus noise. Each β_j is a partial slope — the change in y per unit of x_j with every other predictor held fixed.

Stack n observations into matrices and the whole thing becomes one equation:

y = X·β + ε

Read: y is the n × 1 vector of measurements, each row of the n × (p+1) matrix X is one test run (a leading 1 for the intercept, then that run's predictor values), and β holds the p + 1 coefficients you want.

The least-squares solution:

β̂ = (Xᵀ·X)⁻¹ · Xᵀ·y

Read: the coefficients that minimize the sum of squared misses come from solving a small (p+1) × (p+1) linear system — the same normal equations as a straight-line fit, just bigger.

Coefficient uncertainty:

Var(β̂) = s² · (Xᵀ·X)⁻¹ ,   s² = Σ r_i² / (n − p − 1)

Read: the residual scatter s sets the noise scale, and the diagonal of (XᵀX)⁻¹ says how well your particular test matrix pins down each coefficient. Bad run layouts show up here, not in the fit itself. The formula assumes the noise terms are independent with equal variance; slow drift or run-order correlation in logged data quietly breaks that and makes the quoted standard errors too small — sometimes by a lot.

Model quality, honestly scored:

R²_adj = 1 − (1 − R²) · (n − 1)/(n − p − 1)

Read: adjusted R² charges rent for every predictor you add. Plain R² can only go up as you throw in terms; the adjusted version drops when a new term isn't earning its keep.

Where you meet it

Compensating a transducer on the bench. A pressure transmitter's output depends on pressure and, weakly, on ambient temperature. Log eight runs — 0 to 20 psi at chamber soaks of −10 and +50 °C — and the fit I = 4.016 + 0.7992·P − 0.0023·T (mA, psi, °C) hands you the sensitivity 0.7992 ± 0.0009 mA/psi and the temperature coefficient −0.0023 ± 0.0002 mA/°C in one pass, with residual scatter s ≈ 0.018 mA. That temperature term goes straight into the DAQ compensation table.
The empirical model after a designed experiment. A factorial on a spray-coating cell — gun pressure, standoff, feed rate — ends with a regression: thickness as a function of all three, plus whichever interactions survived. The equation, not the raw runs, is what the process spec cites.
Thrust-stand data reduction. Measured thrust rides on chamber pressure, propellant inlet temperature, and ambient pressure. Regressing thrust on all three separates the term you're selling from the terms the facility imposed on you, which is the difference between a clean data package and an argument at the test review.
Reliability and screening reviews. Time-to-failure (usually logged) against stress, temperature via 1/T, and vibration level — an Arrhenius-style acceleration model is a multiple regression wearing lab clothes, and the review board will ask for the coefficient confidence intervals, not just the fit.

How it works

The machinery is linear least squares with more columns ^[9]. Everything from the two-variable case carries over: the objective is a smooth bowl, the minimum is a linear solve, residuals with an intercept sum to zero, and serious software solves it by QR or SVD rather than forming XᵀX, because forming XᵀX squares the condition number (an X at condition 67 becomes a normal-equations matrix at about 4.5 × 10³).

The partial slope is the whole point — and the main trap. β₁ from the multiple fit is not the slope you'd get regressing y on x₁ alone. The multiple fit credits x₁ only with the part of y that the other predictors can't explain. That's the feature: it's how you estimate a temperature coefficient without running temperature in isolation. It's also the trap: if two predictors moved together during the test — pressure and flow on the same valve schedule, say — the math cannot tell their effects apart, and no amount of data at that same schedule fixes it. Only the test design can.

Collinearity is a design disease with a number attached. For each predictor, regress it on the other predictors; call that fit's R-squared R_j². The variance inflation factor

VIF_j = 1 / (1 − R_j²)

says how much that overlap bloats the coefficient's variance. R_j² = 0.90 gives VIF = 10, meaning the standard error is √10 ≈ 3.2× wider than it would be with orthogonal predictors. Symptoms on the bench: coefficients with absurd magnitudes and opposite signs that "fix" each other, huge standard errors on terms you know matter, and coefficients that swing wildly when you add or drop one run. Orthogonal designs — factorials, in particular — exist precisely to hold every VIF at 1.

Degrees of freedom are a budget. You spend one per coefficient; n − p − 1 is what's left to estimate the noise. Fit 7 coefficients to 8 points and you'll get a gorgeous R² and no idea whether any of it is real. At n = p + 1 the fit goes through every point exactly — R² = 1, information content zero. A defensible rule from the response-surface world: keep several times more runs than coefficients, and treasure replicate runs, because replicates are the only model-free estimate of pure error you will ever have.

R² is a seduction, adjusted R² is a chaperone, residual plots are the truth. With n = 20 runs and p = 5 predictors, R² = 0.90 deflates to R²_adj = 0.864 — fine. But neither number detects a curved trend in the residuals, one high-leverage run dragging the whole surface, or scatter that grows with signal level. Plot residuals against fitted values and against each predictor, every time. Structure in those plots means the model form is wrong, and no summary statistic will confess it. And plot residuals against run order or time as well, because the residual-vs-fitted plot is blind to the most common failure on real test data: serially correlated errors from sensor drift, thermal transients, or back-to-back logged runs. Correlated errors leave the coefficients roughly alone but make the standard errors optimistic — the very numbers the review board asked for. Randomize run order when you can; when the data came in an un-randomized sequence, treat the quoted standard errors as a lower bound.

The fit is a map of where you tested, not of the world. With several predictors, "inside the tested range" means inside the cloud of tested combinations, not inside each variable's individual min and max. If you never ran high pressure together with low temperature, the model has no standing at that corner — even though both values, separately, look interpolated. And a regression on observational data (nobody set the knobs; you logged what happened) supports prediction, not causation: the coefficient on a predictor that merely traveled with the true cause will look just as significant.

History

The engine underneath is least squares, published by Legendre in 1805 and claimed — with earlier use and a full probabilistic theory — by Gauss in 1809 ^[1]^[2]. The word "regression," though, arrived eighty years later from a completely different direction: Francis Galton, studying heredity, noticed that the children of unusually tall parents were tall but less so, and in 1886 published "Regression towards Mediocrity in Hereditary Stature" ^[3]^[4]. For Galton the term described a biological pull toward the average; the name outlived the biology ^[1]^[4].

Turning Galton's one-variable idea into an engineering tool was largely the work of Karl Pearson and, especially, his one-time assistant George Udny Yule in London in the 1890s ^[1]^[5]. Yule's 1897 paper "On the Theory of Correlation" worked out the theory of regression on several variables, and he introduced the partial coefficients that make "holding the others fixed" a computable statement rather than a wish ^[5]^[7]. Then he used it. His 1899 study of English pauperism — pauper rate regressed on the ratio of outdoor to indoor relief, controlling for the proportion of the population over 65 and for population change across Poor Law unions — is regarded as the first application of multiple regression to social observational data, and the method became standard practice almost immediately ^[5]^[6]. It is a fair candidate for the first regression ever run to settle a policy argument, which means the first argument about confounded predictors followed shortly after.

The inference layer engineers now lean on — standard errors, significance tests, and distribution theory for the fitted coefficients — came from R. A. Fisher, whose 1922 paper on the goodness of fit of regression formulas and the distribution of regression coefficients put exact small-sample tests under the method ^[8]. Fisher also loosened the assumptions: Pearson and Yule had leaned on jointly normal variables, while Fisher required only that the response's scatter about the model be well behaved — much closer to how a test engineer actually meets the problem, with the predictor values set by hand and only the response left to chance ^[1]^[8].

Related tools

/tools/loop-4-20ma — the scaling that a multi-term transmitter calibration (span plus temperature compensation) feeds
/tools/strain-gauge-bridge — bridge outputs ride on load and temperature together; separating them is a two-predictor regression
/tools/snr-enob — the acquisition chain's noise floor is the s your coefficient standard errors are built from
/tools/arrhenius-rate — log-rate against 1/T plus other stresses is multiple regression in reliability clothing

Sources

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →