Critical Points and Extrema

When optimizing a machine learning model, our goal is often to find the parameter values that minimize a loss function. These optimal points are typically found at the "bottom" of valleys or the "top" of hills on the loss surface. In calculus, these turning points are known as extrema, and the specific locations where they might occur are called critical points. Understanding how to identify and classify critical points is fundamental to solving optimization problems, from simple curve fitting to complex neural network training.

A critical point of a function $f(x)$ is any point $c$ in the domain of $f$ where the first derivative $f'(c)$ is either zero or undefined. These points are significant because they are the only candidates for local maxima or local minima. Intuitively, if a function is smoothly changing direction from increasing to decreasing (or vice-versa), its tangent line must be horizontal, meaning its derivative is zero. If the function has a sharp corner or a discontinuity, the derivative might not exist at that point, also indicating a potential turning point.

The most common way to find critical points is to compute the first derivative of the function and set it equal to zero. Solving the resulting equation for $x$ will yield the $x$-coordinates of these critical points. This approach works well for polynomial functions and other differentiable functions where the derivative is always defined. These points correspond to where the function's slope momentarily flattens out.

Some functions have critical points where their derivative does not exist. This often occurs at sharp corners (like in the absolute value function), cusps, or points where the tangent line is vertical. These are points where the function's smoothness breaks down, but they still represent potential locations for local extrema. When analyzing a function, always consider its domain and points where its derivative might become undefined, such as denominators becoming zero or arguments of square roots becoming negative.

Once critical points are identified, the First Derivative Test helps classify them as local maxima, local minima, or neither. This test involves examining the sign of the first derivative on either side of a critical point. If the derivative changes from positive to negative, it indicates a local maximum. A change from negative to positive signifies a local minimum. If the derivative's sign does not change, the critical point is typically a saddle point (or an inflection point in 1D), where the function flattens but continues in the same direction.

An alternative method for classifying critical points is the Second Derivative Test. This test uses the concavity of the function at a critical point to determine if it's a local maximum or minimum. If $f''(c) > 0$, the function is concave up, indicating a local minimum. If $f''(c) < 0$, the function is concave down, indicating a local maximum. However, if $f''(c) = 0$, the test is inconclusive, and you must revert to the First Derivative Test.

For a continuous function on a closed interval $[a, b]$, the Extreme Value Theorem guarantees that both an absolute maximum and an absolute minimum exist. These absolute extrema must occur either at a critical point within the interval or at one of the endpoints $a$ or $b$. To find them, you evaluate the function at all critical points in the interval and at the endpoints, then compare these values to find the largest and smallest.

In machine learning, we often train models by minimizing a loss function $L(\theta)$, where $\theta$ represents the model's parameters. The goal is to find the $\theta$ values that yield the smallest possible loss. This is an optimization problem, and the techniques for finding critical points are directly applicable. We seek parameters where the gradient of the loss function, $\nabla L(\theta)$, is zero. These critical points can be local minima (our desired outcome), local maxima, or saddle points. Gradient descent algorithms iteratively move towards these critical points, ideally converging to a local minimum.