Gradient descent

The formula

The whole method is one line, applied over and over:

x_{k+1} = x_k − η · ∇f(x_k)

Reading it: take your current guess x_k, compute the gradient ∇f (the vector that points uphill, steepest-ascent direction), flip the sign to head downhill, scale it by a step size η, and that's your next guess. Repeat until the steps stop moving you.

The gradient is just the stack of partial derivatives:

∇f = [ ∂f/∂x₁ , ∂f/∂x₂ , … , ∂f/∂xₙ ]

Reading it: each component tells you how fast the cost f changes as you nudge that one parameter, holding the rest fixed. Zero gradient means you're at a flat spot — a minimum, a maximum, or a saddle.

The step size η (called the learning rate in machine-learning code) is the one knob you actually tune:

too small  →  crawls, thousands of iterations
too large  →  overshoots the valley, diverges
just right →  monotone decrease every step

Stochastic gradient descent (SGD) swaps the exact gradient for a cheap noisy estimate — the gradient of one data point, or a small batch, instead of the whole dataset:

x_{k+1} = x_k − η · ∇f_i(x_k)     (i drawn at random each step)

Reading it: you trade an exact-but-expensive step for a jittery-but-cheap one, and take a lot more of them. On a million-row dataset that's the difference between one update per pass and thousands.

Where you meet it

Curve fitting on the bench. You've got sensor data and a model with parameters — a thermistor's Steinhart-Hart coefficients, a diode's I-V curve, a nonlinear spring rate. Least-squares fit means minimizing sum of squared residuals, and when the model isn't linear in its parameters, the solver underneath is a cousin of this method: Levenberg-Marquardt blends the gradient-descent direction with Newton's method, leaning toward steepest descent when it's far from the solution and switching to the much faster Gauss-Newton step once it's close. So your curve_fit call in Python shares this entry's gradient machinery, but it isn't running the plain x − η∇f loop.
Sensor and instrument calibration. Fitting a polynomial or spline to map raw counts to engineering units, or backing out the six-plus terms of an accelerometer or load-cell calibration, is an optimization. The cost is calibration error across your reference points; the parameters are the correction coefficients. Gradient descent is one of the engines that lands them.
Every ML training loop at the review board. When the software team says the anomaly-detection model or the vision system "is training," SGD is the loop running. The cost is prediction error over the training set, the parameters are the network weights — sometimes billions of them — and there's no closed-form solution, so the only move is to step downhill a few hundred thousand times.
Controller and filter tuning. Auto-tuning a PID gain set, or fitting a Kalman filter's noise covariances to logged flight data, both reduce to minimizing a tracking-error cost over a handful of parameters. Gradient methods handle it when the error surface is smooth enough to differentiate.

How it works

The mental picture is a ball on a hillside. The gradient is the direction of steepest slope; you roll opposite it. On a bowl-shaped (convex) cost with a well-chosen step, you converge to the one true minimum. That's the good case, and it's why the method has survived 175 years.

The trouble is the shape of the bowl. Take a simple quadratic cost f(x,y) = ½(x² + a·y²). When a = 1 the level sets are circles and gradient descent walks straight to the bottom. When a = 10 they're stretched ellipses — a steep-walled canyon — and the gradient points mostly across the canyon, not down it. Set the step to the best fixed value and the iterates zig-zag down the valley, the distance to the minimum shrinking by only (κ−1)/(κ+1) per step, where κ is the condition number. For κ = 10 that's 0.818 — and since the cost is quadratic in that distance, the cost shrinks by its square, 0.67 per step: verified numerically, 55 → 36.8 → 24.6, and ~40 steps to reach 10⁻⁵. Push κ to 1000 and the factor is 0.998; you'll wait forever. Ill-conditioning is the number-one reason gradient descent crawls, and it's a property of your problem, not your code.

The step size is the classic self-inflicted wound. In that same a = 10 bowl, any η above 2/L (here 2/10 = 0.2) diverges — set η = 0.25 and the stiff coordinate grows 1.5× every step: after 10 steps it's ~58, after 20 it's ~3300, and it never comes back. Too small and you're paying for iterations that barely move. The whole cottage industry of Adam, RMSProp, momentum, and learning-rate schedules exists to dodge this. Momentum — Polyak's heavy-ball idea — adds inertia so the iterate coasts down the valley instead of ping-ponging across it, and it's usually the cheapest fix for the zig-zag.

The other limit of validity: gradient descent only finds a local minimum. On a non-convex surface — which is every neural network and most nonlinear fits — where you land depends on where you start. It also stalls at saddle points where the gradient is zero but you're not at a minimum. And it needs a differentiable cost; a hard threshold or an abs() with a kink will hand you a gradient of zero or a discontinuity right where it matters.

SGD adds a twist worth understanding. The noisy single-sample gradient is wrong on any given step, but right on average, and the noise is a feature: it kicks the iterate out of shallow local minima and saddles that would trap the exact method. The cost is that you can't use a fixed step forever — the classic Robbins-Monro condition says the steps must shrink (Σηₖ = ∞ but Σηₖ² < ∞, e.g. ηₖ ∝ 1/k) for the noise to average out and the thing to actually converge. The mistake people make: leaving the learning rate high and wondering why the loss curve rattles around a floor instead of settling. It's not stuck — it's still stepping, just too big.

History

Augustin-Louis Cauchy — the same Cauchy from complex analysis and the Cauchy sequence — wrote the method down in an 1847 note to the French Academy of Sciences, Méthode générale pour la résolution des systèmes d'équations simultanées ^[1]^[2]. He wasn't chasing machine learning; he was trying to solve systems of equations that came up in astronomy and orbital calculations ^[5], and the observation was almost offhand: a function decreases, at least at first, if you step along the negative of its gradient ^[1]. That one sentence is the whole method. He gave no convergence proof and no step-size rule — those came later.

Jacques Hadamard sketched a similar idea in 1907, and Haskell Curry was the first to seriously study when the method actually converges for nonlinear problems, in 1944 ^[2]. The stochastic branch — the one that powers modern ML — traces to Herbert Robbins and Sutton Monro's 1951 paper A Stochastic Approximation Method in the Annals of Mathematical Statistics ^[3]^[4]. They were solving a root-finding problem with noisy measurements, not training a model, but their procedure is exactly SGD viewed from the other side: replace the true gradient with a crude random estimate and it still converges, provided the steps shrink on their schedule ^[3]. Jack Kiefer and Jacob Wolfowitz published a closely related optimization version a year later. The line from Cauchy's astronomy note to a GPU training a billion-parameter network is unbroken, and it's still the same line of arithmetic at the core.

Related tools

/tools/strain-gauge-bridge
/tools/loop-4-20ma
/tools/vibration-natural-freq
/tools/q-factor-bandwidth
/tools/reynolds-number

Sources

https://www.osti.gov/servlets/purl/983240
https://en.wikipedia.org/wiki/Gradient_descent
https://en.wikipedia.org/wiki/Stochastic_gradient_descent
https://www.scirp.org/reference/referencespapers?referenceid=2228998
C. Lemaréchal, "Cauchy and the Gradient Method", Documenta Mathematica, Extra Volume ISMP (2012), 251–254 — https://ems.press/content/book-chapter-files/27368

Written by HE in our own words from the cited sources — engineering judgment included, your stamp still required. All entries →