Shannon entropy

The formula

H = −Σ pᵢ · log₂(pᵢ)

Take every possible outcome, weight the log of its probability by that probability, sum, and flip the sign. The result is the average information per symbol, in bits, from a source with outcome probabilities p₁ … pₙ.

I(x) = −log₂(p(x)) = log₂(1/p(x))

The information content of one specific outcome. Rare events carry more bits: a 1-in-1,024 event carries 10 bits, a sure thing carries zero. Entropy is just the expected value of this quantity.

0 ≤ H ≤ log₂(N)

The bounds. Entropy is zero when one outcome is certain, and maxes out at log₂(N) when all N outcomes are equally likely. A fair coin gives exactly 1 bit per flip; a fair die gives log₂(6) ≈ 2.585 bits per roll.

1 nat = 1/ln(2) ≈ 1.443 bits

Change the log base and you change the unit, nothing else. Base 2 gives bits, base e gives nats, base 10 gives hartleys. Physics and ML papers run in nats; comms and storage run in bits.

Where you meet it

Telemetry data-rate budgets. A 16-bit channel sampled at 10 kS/s is 160 kbit/s raw. If the signal's actual entropy rate is 6 bits per sample, the lossless floor is 60 kbit/s and no compressor, present or future, averages below it. When the review board asks whether the recorder can be halved by "better compression," entropy is the number that answers before the vendor does.
On the bench, judging a compression claim. Run the candidate data through a decent general-purpose compressor. If it barely shrinks, the data is near its entropy already — usually because the low-order ADC bits are noise, and noise is incompressible by definition. A vendor promising 10:1 lossless on wideband sensor data is promising to beat Shannon; the meeting can end there.
Random number and key generation. Every hardware RNG qualification comes down to entropy per output bit. NIST SP 800-90B builds its whole assessment around this — though on the min-entropy variant, not Shannon's average (see the gotcha below) ^[10]^[11].
Instrument and bus sizing. An event flag that's set 1 time in 10 carries 0.47 bits per report, not 1. Status words, discretes, and health telemetry are almost all redundancy, which is exactly why event-driven reporting and delta encoding buy back so much bus bandwidth.

How it works

Entropy measures surprise, averaged. An outcome you saw coming tells you nothing; an outcome that shocks you tells you a lot; entropy is the long-run average over the source's statistics. Shannon's source coding theorem turns that into hardware truth: a source of entropy H bits/symbol can be losslessly encoded at any rate above H, and at no rate below it ^[1]^[2]. It is a two-sided result, floor and achievability at once — the same structure as its sibling, channel capacity, which caps the other end of the link.

Behavior worth internalizing:

Skew collapses entropy fast. A 90/10 binary source carries 0.469 bits per symbol, not 1. At 99/1 it's down to 0.081 bits. Real engineering data — status flags, quiet strain gauges, frame headers — is heavily skewed, which is why it compresses so well and why raw bit width says nothing about information content.
Correlation collapses it further. The per-symbol formula assumes independent symbols. Real signals have memory — the next sample of a 100 Hz vibration channel is largely predictable from the last few — so the entropy rate of the sequence sits well below the entropy of one sample viewed alone. This gap is precisely what predictive and delta coders harvest. Uniform 26-letter text would be 4.70 bits per letter; Shannon measured actual English at roughly 0.6 to 1.3 ^[1]^[6].
Entropy lives in the model, not the file. "The entropy of this file" is not a defined quantity until you say what probability model generated it. A compressor's output size is an estimate of entropy under that compressor's implicit model, and a better model gives a lower number. This is why two tools report different "entropy" for the same log file and neither is lying.

The mistakes people actually make:

Reading entropy as importance. A dead-flat noise channel has maximum entropy and zero engineering value. Entropy counts bits of unpredictability; it does not know which bits you care about. Budgeting a downlink by entropy alone will faithfully preserve the noise.
Using Shannon entropy where min-entropy is required. Security lives on the worst case, not the average. Min-entropy, H_min = −log₂(p_max), is what an attacker guessing the most likely output faces: for the 90/10 source it's 0.152 bits against Shannon's 0.469. Quote Shannon entropy for a key generator and you overstate its strength by 3× in that example — which is why the standards body doesn't let you ^[10]^[11].
Sliding into the continuous case carelessly. Differential entropy — the integral analog with a density in place of the pᵢ — can be negative, changes value under a units change, and is not "the entropy of the analog signal." Quantize a continuous source at step Δ and the discrete entropy grows like log₂(1/Δ) without bound. Bits only become meaningful after you fix a quantizer, which is an engineering decision, not a property of the waveform.
Choking on the zero. 0 · log₂(0) is taken as 0 — the limit, and the convention. Impossible outcomes contribute nothing. Code that computes entropy without guarding p = 0 returns NaN, and this exact bug ships constantly.

History

The logarithmic measure came first, without the probabilities. In 1928, Ralph Hartley at Bell Labs proposed quantifying information as the log of the number of distinguishable messages — fine for a source where every message is equally likely, silent about the ones that aren't ^[7]^[8]. Twenty years later Claude Shannon, at Bell Labs since 1941, closed that gap. His 1948 paper "A Mathematical Theory of Communication" in the Bell System Technical Journal weighted the log by probability, proved the resulting H was the unique measure satisfying a short list of reasonable axioms, and built source coding, channel capacity, and effectively the entire field on top of it ^[1]^[2]^[3]. The paper is also where the word "bit" entered print — Shannon credits it as "a word suggested by J. W. Tukey," who had contracted "binary digit" in a Bell Labs memo the year before ^[2]^[4].

The name is the good story. Shannon's formula has the same shape as the entropy of statistical mechanics, and Shannon said so in the paper itself, pointing at Boltzmann's H-theorem ^[2]. A 1971 Scientific American article by Myron Tribus and E. C. McIrvine added the anecdote everyone repeats: John von Neumann told Shannon to call it entropy — partly because the mathematics already carried that name, and partly because nobody knows what entropy really is, so in a debate he would always have the advantage ^[1]^[5]. The line is a decades-later recollection, so hold it loosely; the mathematical kinship, at least, is right there in Shannon's own text.

Shannon then did the experiment engineers still find charming: in 1951 he had people guess English text letter by letter and used their hit rate to bound the language's entropy at roughly 0.6 to 1.3 bits per letter — meaning English is about three-quarters redundant, which is why your compressed tarball of documentation is a quarter the size of the original ^[6]^[9].

Related tools

Sources

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →