An iterative optimization method that updates variables by moving in the direction opposite to the gradient of an objective function to reduce its value, typically using step-size (learning rate) control.

Principle

The organizing idea is that the negative gradient is the direction of steepest local decrease of a differentiable objective, so successive small moves along −∇f reduce the function value until a stationary point is reached under suitable conditions.

Demonstration

Minimize a convex quadratic f(x)=1/2 x^T A x − b^T x with A symmetric positive definite by iterating x_{k+1} = x_k − α ∇f(x_k) = x_k − α(Ax_k − b); with α chosen in (0,2/λ_max(A)) the method converges linearly to the minimizer.

Misapplication

Using an excessively large fixed step size can cause divergence or oscillation; applying plain gradient descent to ill-conditioned problems yields very slow convergence; ignoring nonconvexity can trap iterates at saddle points or poor local minima.

Consequence

When applied with appropriate step-size selection and problem regularity, gradient descent converges to stationary points and provides a simple, scalable algorithm for large-scale optimization and machine learning.

Reversal

The opposite notion is gradient ascent, which moves along the gradient to increase the objective; more drastically, second-order or quasi-Newton methods replace the local steepest direction with curvature-informed search directions.

Boundary

Requires differentiability (or subgradient-generalizations) of the objective and appropriate Lipschitz or convexity conditions for global guarantees; excludes problems with only discrete decision variables unless relaxed.

Semantic Tension

Competes with coordinate descent, stochastic methods, and Newton-type methods: gradient descent is simple and memory-light but can be slower than curvature-aware methods or faster stochastic variants in large-data regimes.

Synthesis

Gradient descent is the prototypical first-order descent algorithm: compute the local gradient, choose a step length consistent with stability and progress, and iterate to reduce the objective, trading simplicity and scalability against possible slow convergence on ill-conditioned or nonconvex landscapes.