Gradient Descent

Click on the contour plot to set a starting point, then step or run the iteration.

Heat map & iso-contours of \(f\). Dots = iterates, arrow = negative gradient \(-\nabla f\).

Objective function \(f(x,y)\)

Learning rate \(\eta\) = 0.100

log-scaled slider, \(\eta \in [10^{-4},\,10^{0}]\)

Speed = 1× steps/frame

iteration \(n\)0

point \((x_n, y_n)\)—

\(f(x_n, y_n)\)—

\(\|\nabla f\|\)—

statusidle

The method

To minimize a differentiable function \(f:\mathbb{R}^2\to\mathbb{R}\), gradient descent moves in the direction of steepest decrease, the negative gradient:

\[ \mathbf{x}_{n+1} \;=\; \mathbf{x}_n \;-\; \eta\,\nabla f(\mathbf{x}_n), \qquad \nabla f = \left(\frac{\partial f}{\partial x},\, \frac{\partial f}{\partial y}\right). \]

The step size \(\eta > 0\) (the learning rate) controls how far each step travels. Things to observe with the demo:

Bowl: well-conditioned, so iterates head almost straight to the minimum at the origin.
Ill-conditioned bowl: elongated level sets force the path to zig-zag across the valley, because \(-\nabla f\) points mostly across the valley rather than along it.
Rosenbrock: a curved narrow valley with minimum at \((1,1)\); convergence is notoriously slow.
Himmelblau: four local minima — the limit point depends on where you start.
Divergence: if \(\eta\) is too large the iterates overshoot and blow up. For a quadratic \(f\) with Hessian eigenvalues \(\lambda_i\), convergence requires \(\eta < 2/\lambda_{\max}\).