Covariance and correlation

The formula

Covariance is the average product of two signals after you subtract each one's own mean:

cov(x,y) = ( Σ (x_i − x̄)·(y_i − ȳ) ) / (n − 1)

Reading: when x is above its mean at the same points y is above its mean, the products are positive and pile up — positive covariance. When one runs high while the other runs low, the products go negative. The trouble is that covariance carries units (volts·°C, psi·seconds) and scales with how big your numbers are, so its magnitude tells you almost nothing on its own.

The Pearson correlation coefficient fixes that by dividing out both standard deviations:

r = cov(x,y) / ( s_x · s_y )

Reading: r is covariance rescaled to be dimensionless and pinned to the range −1 ≤ r ≤ +1. It is the covariance you'd get if you first converted both variables to z-scores (mean 0, standard deviation 1). r = +1 means the points sit exactly on an upward line, −1 an exactly downward line, 0 no linear trend at all.

Fully expanded, with nothing hidden:

r = Σ (x_i − x̄)·(y_i − ȳ)
    ───────────────────────────────────────────
    √( Σ (x_i − x̄)² ) · √( Σ (y_i − ȳ)² )

The (n−1) factors cancel top and bottom, which is why r doesn't care whether you used the sample or population form of variance.

Spearman's rank correlation, ρ (rho), is the exact same Pearson formula run on the ranks of the data instead of the values:

ρ = Pearson r computed on rank(x) and rank(y)

With no tied ranks it reduces to a hand-computable form, where d_i is the rank difference at each point:

ρ = 1 − ( 6 · Σ d_i² ) / ( n·(n² − 1) )

Reading: ρ asks whether the two variables move together monotonically — consistently up-with-up — without demanding that the relationship be a straight line.

Where you meet it

Bench characterization. You sweep an amplifier's gain versus supply voltage and want a single number for how coupled they are before you write the sensitivity spec. r off the logged sweep gives you that in one line of code, and its sign tells you which way the coupling runs.
Test-stand data reduction. A thrust-stand run logs chamber pressure, load-cell force, and three thermocouples at 1 kHz. Building the correlation matrix — every channel's r against every other — is the fastest way to spot that a "thermal" channel is actually tracking pressure because a transducer is picking up structural strain, not heat.
RF and noise work. Two receiver channels that should be independent show r = 0.4 in the noise floor. That's not physics, that's a shared ground or a common LO leaking in. Correlated noise between channels is the fingerprint of a coupling path you haven't found yet.
Design-review pushback. Someone puts up a scatter of failures versus lot number, quotes r = 0.7, and asks for a process change. Your job on the board is to ask whether that r survives Anscombe's problem below — and whether anyone has confused correlation with cause.

How it works

The mechanics are simple; the traps are where careers get made. Four things to keep straight.

r only sees straight lines. Take y = x² sampled symmetrically from −3 to +3. The dependence is perfect — y is completely determined by x — yet r computes to exactly 0.0. Covariance and Pearson r measure linear association only. A correlation of zero means "no linear trend," never "no relationship." If you expect a curve, either linearize it first or reach for a rank or mutual-information method.

One number cannot describe a cloud. Anscombe's quartet is four little datasets built to make this unforgettable: all four have the same means, the same variances, and the same correlation — r ≈ 0.816 — while one is a clean line, one is a smooth curve, one is a line dragged off by a single outlier, and one is a vertical stack rescued by one lonely point. Verified numerically, the four r values come out 0.816, 0.816, 0.816, and 0.817. Plot the data. Always plot the data. r is a summary, and summaries lie by omission.

Outliers own the number. Because r squares deviations, one bad point far from the pack can manufacture a correlation that isn't there, or erase one that is. This is exactly why Spearman exists. Run Pearson and Spearman on y = x³ over x = 1…7: Pearson gives r = 0.9344 because the cubic curve isn't a straight line, but Spearman gives ρ = 1.000 because the ranks march in perfect lockstep. When your data is ordinal, heavy-tailed, or has a few gremlins you can't justify deleting, ρ is the more honest answer.

Correlation is not causation — and this is the whole point of the topic. r measures co-movement. It says nothing about which variable drives which, or whether a third, unmeasured thing drives both. Ice-cream sales and drowning deaths correlate strongly; the driver is summer. On a test stand, two channels correlating because they share a power supply is a wiring finding, not a physics finding. The discipline is: a correlation opens an investigation, it never closes one. Before you spend money changing a process on the strength of an r, you owe the review board a mechanism.

A few housekeeping limits. r needs at least a handful of points to mean anything — with n = 3 you can hit r = 1 by luck. It assumes the pairing is real (same event, same timestamp); correlate two misaligned time series and you'll measure your alignment error, not the physics. And a high r on training data is not a validated model; it's a starting hypothesis.

History

The idea grew out of heredity, not engineering. Francis Galton — Darwin's cousin, and a man who measured everything he could reach — spent the 1870s and 1880s trying to quantify how strongly a trait in parents predicted the same trait in offspring. In a paper read to the Royal Society in December 1888, "Co-relations and their Measurement, chiefly from Anthropometric Data," he introduced the concept and the word, defining two organs as "co-related" when variation in one is accompanied on average by variation in the other in the same direction ^[1]^[2]. Galton had the intuition and a hand-fitted slope; he did not yet have the clean formula.

That fell to Karl Pearson, who took Galton's insight and gave it rigor. In an 1896 paper in the Philosophical Transactions of the Royal Society, "Mathematical Contributions to the Theory of Evolution. III. Regression, Heredity and Panmixia," Pearson derived the product-moment coefficient we still use and showed it was the estimate that minimized squared prediction error ^[3]^[4]. His name stuck to it. As often happens in statistics, the underlying math had been written down earlier — the French astronomer Auguste Bravais had derived related product-moment formulas in the 1840s without recognizing what he had ^[3]^[4] — a tidy example of Stigler's law, which holds that discoveries rarely carry the name of their first discoverer.

The rank version came from a different corner. Charles Spearman, a psychologist and former army officer still finishing his doctorate at Leipzig, published "The Proof and Measurement of Association between Two Things" in the American Journal of Psychology in 1904 ^[5]^[6]. Working with human ratings — inherently ordinal, inherently noisy — he built a correlation that used ranks instead of raw scores, giving the world a tool that shrugs off outliers and curved-but-monotonic relationships. Three men, three decades, one habit of mind: measure the thing everyone talks about but nobody has pinned down.

Related tools

/tools/thermal-noise-floor
/tools/friis-cascade
/tools/noise-figure-temperature

Sources

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →