Floating-point arithmetic

The formula

A floating-point number is stored as sign, a fraction (the significand or mantissa), and an exponent. For normal numbers — the overwhelming majority of what you compute with — the IEEE 754 value is:

x = (−1)^s · (1.f) · 2^(e − bias)

Reading: a sign bit, a fraction f that fills in the digits after an implied leading 1, and an exponent e that slides the binary point. It is binary scientific notation with a fixed bit budget. Two exponent patterns are held back from this formula: an all-zeros exponent field encodes zero and the subnormals (where the implied leading digit is 0, not 1), and an all-ones field encodes Inf and NaN — see the special values below.

The two formats you meet daily split those bits like this:

single (float32): 1 sign +  8 exponent + 23 fraction  = 32 bits,  bias 127
double (float64): 1 sign + 11 exponent + 52 fraction  = 64 bits,  bias 1023

Reading: single carries 23 fraction bits, double carries 52. That fraction budget is the whole story of how much precision you get.

The precision itself is one number, machine epsilon — the gap between 1.0 and the next representable value:

ε_single = 2^−23 ≈ 1.19×10⁻⁷      (about 7 decimal digits)
ε_double = 2^−52 ≈ 2.22×10⁻¹⁶     (about 16 decimal digits)

Reading: this is the smallest relative step the format can resolve near 1.0. Within the normal range, and under the default round-to-nearest mode, every value is stored to within a factor of 1 ± ε/2 of the truth. (The guarantee loosens in the subnormal range, where absolute rather than relative error is what's bounded, and fails entirely at overflow.)

The step size is relative, so the absolute gap grows with magnitude:

gap between adjacent floats near x  ≈  ε · 2^floor(log₂ |x|)

Reading: the number line isn't evenly spaced. Floats are dense near zero and spread out as they get big. That single fact explains most of the surprises below.

Where you meet it

The 0.1 + 0.2 moment at your desk. Type 0.1 + 0.2 into Python, MATLAB, or a C double and you get 0.30000000000000004, not 0.3. Neither 0.1 nor 0.2 has a finite binary expansion — same way 1/3 has no finite decimal — so the stored values are already slightly wrong, and the error survives the add. An equality test if (a + b == 0.3) fails, and a junior engineer spends an afternoon on it.
A float32 sensor log that stops counting. You stream a 32-bit accumulator or a sample index into a float and it climbs fine — at first. Above 2²³ = 8,388,608 the gap between adjacent float32 values is 1.0, so integer counts are still exact but nothing in between survives. At 2²⁴ = 16,777,216 the gap doubles to 2.0: adding 1 lands exactly halfway between representable values and rounds back down (round-half-to-even), so the counter freezes at exactly 16,777,216 while the raw data keeps coming. This bites data acquisition on long-duration test-stand runs and any high-rate telemetry logged in single precision.
A test report where two people's numbers don't match. One analyst summed a vibration record in float32, another in float64, and they disagree in the fourth digit at the review board. Neither is wrong about the physics; they lost different amounts of the sum to rounding. The double-precision result is the one to trust, and "what precision did you accumulate in" becomes a real review question.
A control loop that returns NaN and takes the whole thing down. A guidance or filter computation divides 0/0, takes sqrt of a slightly-negative number that should have been zero, or subtracts two Infs. The result is NaN, and NaN is contagious — every downstream multiply and add produces NaN too, so one bad operation silently poisons an entire state vector until something trips a limit.

How it works

The core bargain: floating point trades exactness for range. You get to represent numbers from roughly 10⁻³⁸ to 10³⁸ in single, or 10⁻³⁰⁸ to 10³⁰⁸ in double, but only about 7 or 16 significant digits at any magnitude. It is a relative-error machine. That is the right model for physical quantities, which you usually know to a few digits anyway — and exactly the wrong model for exact integer counts, money, or anything where the last digit is load-bearing.

The special values are features, not accidents. IEEE 754 reserves the exponent patterns the normal-number formula leaves out for:

+Inf, −Inf   — overflow, or a finite number divided by zero (1.0/0.0 = +Inf)
NaN          — an operation with no sensible answer: 0/0, Inf−Inf, sqrt(−1), log(−1)
±0           — signed zero, so 1/(+0)=+Inf and 1/(−0)=−Inf stay distinguishable
subnormals   — numbers below the smallest normal, stored as (0.f)·2^(1−bias),
              giving "gradual underflow" instead of a cliff straight to zero

Inf and NaN mean a computation can keep running and flag trouble rather than crashing on the spot. The catch is that NaN propagates: any arithmetic touching a NaN yields NaN, and — the part that trips people — NaN == NaN is false. The standard way to test for one is x != x.

The mistakes that actually cost time:

Testing floats for exact equality. a == b on computed floats is almost always a bug. Compare against a tolerance instead: abs(a − b) <= tol, sized to the magnitudes involved (a few ε times the larger operand is a sane start).
Catastrophic cancellation. Subtract two nearly-equal large numbers and the digits that agreed cancel out, leaving mostly rounding noise where your answer used to be. It shows up in the quadratic formula when b² ≫ 4ac, in a near-balanced bridge, in differencing two GPS positions. The fix is algebraic — rearrange so the subtraction never happens — not a bigger float.
Summing a long series naively. Add a million small numbers to a growing total and each addition rounds off the small one against the big running sum; the error piles up. Kahan summation carries a running correction term and recovers most of the lost bits for a couple extra flops. Reach for it on long accumulations.
Assuming associativity. (a + b) + c need not equal a + (b + c) in floating point. Compilers reorder at their peril, and two "identical" codes can give different last digits. This is why bit-exact reproducibility across machines is hard.

Limit of validity worth taping to the monitor: single precision gives about 7 good decimal digits, double about 16. If your problem needs more digits than that after the amplification of your algorithm, no library call saves you — you change the math or the precision, deliberately.

History

The mess that IEEE 754 cleaned up was real: through the 1970s every vendor's hardware rounded differently, handled overflow differently, and disagreed on what 0/0 should do, so a program's answers changed when you moved it between machines. In 1976 John Palmer at Intel set out to fix that for the coming 8087 math coprocessor and brought in William "Velvel" Kahan, a numerical analyst at Berkeley, to design the arithmetic ^[1]^[2]. Kahan pulled in Harold Stone and his graduate student Jerome Coonen, and the "KCS draft" they wrote became the working proposal for the IEEE subcommittee that formed in 1977 ^[1].

Kahan's design carried the ideas that make the standard humane to work with: infinities so a divide-by-zero flags instead of crashing, NaN so a nonsense operation reports itself rather than returning a plausible lie, subnormal numbers for gradual underflow near zero, and a balanced exponent bias ^[1]^[2]. The Intel 8087, announced in 1980, was the first chip to implement the draft, so the standard shipped in silicon years before it was law ^[2]^[3]. IEEE 754 was formally approved in 1985 and adopted internationally as ISO/IEC 60559 in 1989 — eight years from the subcommittee's founding ^[1]. Kahan received the ACM Turing Award in 1989 for the work, and is widely called the father of floating point ^[4]^[3].

The field learned the cost of the alternative the hard way. On 25 February 1991, a Patriot battery at Dhahran failed to intercept an incoming Scud that struck a U.S. Army barracks, killing 28 soldiers. The U.S. General Accounting Office traced it to a timing calculation: the system multiplied its clock by 0.1 to convert tenths of a second into seconds, and 0.1 has no exact finite binary representation, so the chopped constant accumulated error ^[5]. After about 100 hours of continuous operation the clock had drifted roughly 0.34 seconds — far enough, at Scud speed, to put the target outside the tracking gate ^[5]. The specific hardware there was a 24-bit fixed-point register rather than IEEE floating point, but the root cause is the identical one every engineer meets in 0.1 + 0.2: a friendly-looking decimal that binary cannot store exactly.

Related tools

/tools/adc-resolution
/tools/snr-enob
/tools/db-converter
/tools/rms-peak

Sources

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →