# Maxima, minima, and saddle points

Learn what local maxima/minima look like for multivariable function.

## Background

## What we're building to

- Intuitively, when you're thinking in terms of graphs, local maxima of multivariable functions are peaks, just as they are with single variable functions.
- The gradient of a multivariable function at a maximum point will be the zero vector, which corresponds to the graph having a flat tangent plane.
- Formally speaking, a local maximum point is a point in the input space such that all other inputs in a small region near that point produce smaller values when pumped through the multivariable function .

## Optimizing in higher dimensions

One of the most important applications of calculus is its ability to sniff out the maximum or the minimum of a function.

- Perhaps you find yourself running a company, and you've come up with some function to model how much money you can expect to make based on a number of parameters, such as employee salaries, cost of raw materials, etc., and you want to find the right combination of resources that will maximize your revenues.
- Maybe you are designing a car, hoping to make it more aerodynamic, and you've come up with a function modelling the total wind resistance as a function of many parameters that define the shape of your car, and you want to find the shape that will minimize the total resistance.
- In machine learning and artificial intelligence, the way a computer "learns" how to do something is commonly to minimize some "cost function" that the programmer has specified.

## Local maxima and minima, visually

Let's start by thinking about those multivariable functions which we can graph: Those with a two-dimensional input, and a scalar output, like this:

I chose this function because it has lots of nice little bumps and peaks. We call one of these peaks a

**local maximum**, and the plural is**local maxima.**- The point underneath a peak in the input space (which in this case means the -plane) is called a
**local maximum point**. - The output of a function at a local maximum point, which you can visualize as the height of the graph above that point, is the
**local maximum**itself.

The word "local" is used to distinguish these from the

**global maximum**of the function, which is the single greatest value that the function can achieve. If you are on the peak of a mountain, it's a local maximum, but unless that mountain is Mt. Everest, it is not a global peak.I'll give you the formal definition of a local maximum point at the end of this article. Intuitively, it is a special point in the input space where taking a small step in any direction can only decrease the value of the function.

Similarly, if the graph has an inverted peak at a point, we say the function has a

**local minimum point**at the value above/below this point on the -plane, and the value of the function at this point is a**local minimum**. Intuitively, these are points where stepping in any direction can only increase the value of the function.## Stable points in one variable (review)

You may remember the idea of local maxima/minima from single-variable calculus, where you see many problems like this:

**Concept check**: For what value is the function the greatest? What is the maximum value?

In general, local maxima and minima of a function are studied by looking for input values where . This is because as long as the function is continuous and differentiable, peaks and valleys will flatten out, in that the tangent line at a local maximum or minimum has slope .

Such a point has various names:

- Stable point
- Critical point
- Stationary point

All of these mean the same thing:

The requirement that be continuous and differentiable is important, for if it was not continuous, a lone point of discontinuity could be a local maximum:

And if is continuous but not differentiable, a local maximum could look like this:

In either case, talking about tangent lines at these maximum points doesn't really make sense, does it?

However, even when is continuous and differentiable, it is not enough for the derivative to be , since this also happens at inflection points:

This means finding stable points is a good way to start the search for a maximum, but it is not necessarily the end.

## Stable points in two variables

The story is very similar for multivariable functions. When the function is continuous and differentiable,

**all the partial derivatives will be at a local maximum or minimum point**.With respect to the graph of a function, this means its tangent plane will be flat at a local maximum or minimum. For instance, here is a graph with many local extrema and flat tangent planes on each one:

Saying that all the partial derivatives are zero at a point is the same as saying the gradient at that point is the zero vector:

People often write this more compactly like this:

The convention is that

**bold**variable are vectors. So is a vector of the input values and is the vector with all zeros.Such an input goes by the same various names as in the single-variable case:

- Stable point
- Stationary point
- Critical point

The thinking behind the words "stable" and "stationary" is that when you move around slightly near this input, the value of the function doesn't change significantly. The word "critical" always seemed a bit over dramatic to me, as if the function is about to die near those points.

As with single variable functions,

**it is not enough for the gradient to be zero**to ensure that a point is a local maximum or minimum. For one thing, you can still have something similar to an inflection point:But there is also an entirely new possibility, unique to multivariable functions.

## Saddle points

Consider the function . Let's make a few observations about what goes on around the origin

- Both partial derivatives are at this point:

Therefore is a stable point.

- When you just move in the direction around this point, the function looks like . The single-variable function has a
**local minimum**at . - When you just move in the direction around this point, meaning the function looks like . The single-variable function has a
**local maximum**at .

In other words, the and directions disagree over whether this input should be a maximum or a minimum point. So even though is a stable point, and is not an inflection point, it cannot be a local maximum or local minimum!

Here's a video of this graph rotating in space:

Doesn't the region around kind of have the shape of a horse's saddle?

Well, mathematicians thought so, and they had one of those rare moments of deciding on a good name for something:

**Saddle points**. By definition, these are stable points where the function has a local maximum in one direction, but a local minimum in another direction.## Testing maximality/minimality

"Alright,"

I hear you saying,

"so it's not enough for the gradient to be since you might have an inflection point or a saddle point. But how can you tell if a stable point is a local maximum or minimum?"

I'm glad you asked! This is the topic of the next article on the second partial derivative test. For now, let's finish things off with a formal definition of a local maximum.

## Formal definition

I've said this before, but the reason to learn formal definitions even when you already have an intuition to expose yourself to how intuitive mathematical ideas are captured precisely. It's good practice for thinking clearly, and it can also help to understand those times when intuition differs from reality.

In defining a local maximum, let's use vector notation for our input, writing it as .

**Formal definition of a local maximum**: A scalar-valued function has a local maximum at if there exists some positive number , thought of as a radius, such that the following statement is true:

That's a bit of a mouthful, so let's break it down:

Saying "" means the variable is within a distance of the maximum point . When is two-dimensional this is the same as saying lies inside a circle of radius centered at the point .

More generally, if is -dimensional, the set of all such that forms an -dimensional ball with radius centered at .

We can then translate this definition from math-speak to something more closely resembling English as follows:

- is a maximum point of if there is some small (ball-shaped) region in the input space around the point such that the highest possible value you can get for evaluated on points in that region is achieved at the point .

**Test your understanding**: Write the formal definition for a local minimum, and think about what each component means as you write it down. (Resist the temptation to just copy down the words in the definition above.)

## Summary

- Intuitively, when you're thinking in terms of graphs, local maxima of multivariable functions are peaks, just as they are with single variable functions.
- The gradient of a multivariable function at a maximum point will be the zero vector, which corresponds to the graph having a flat tangent plane.
- Formally speaking, a local maximum point is a point in the input space such that all other inputs in a small region near that point produce smaller values when pumped through the multivariable function .