When reading this article I noticed the "Subject: Prometheus" button up at the top just to the right of the KA homesite link. Anyone else notice this?

Nope. I have a "Subject: Multivariable Calculus" button. Glitch?

what R should be? any value? does the limit of R tends to zero?

The specific value of r is situational, depending on how "local" you want your max/min to be.

f(x)≥f(x0) why it is allowed to be greater or EQUAL ? isn't it just greater?

Well think about what happens if we do what you are suggesting. We cant have the point x = x0 then yet when we say for all x we mean for the entire domain of the function. x0 thus must be part of the domain if we are able to evaluate it in the function. So what happens when x does equal x0? If the definition was just > and not >= then we would find that the condition is not true and thus the point x0 would not be a maximum which is not what we want.

Saddle point and inflection point are two things I have a hard time differentiating! They are not the same, graphically, but what are the key mathematical differences between them both?

Saddle points are used for multivariable functions, while inflection points are found in the case of single variable functions. So, that's one point of difference. As for what they are, saddle points are extremum points which are neither local minima or maxima (How are they extrema then? Well, they show up when you try to find extrema, but once you use the second derivative test on them, they return 0) Inflection points are points where functions change concavity (go from increasing to decreasing or vice versa).

_In machine learning and artificial intelligence, the way a computer "learns" how to do something is commonly to minimize some "*cost function*" that the programmer has specified._ I think it should be *loss function*--not *cost function*?

Well, if doing A costs B, then by doing A you lose B. So, whether you call it cost function or loss function is just semantics – the meaning is the same.

Main content

Course: Multivariable calculus > Unit 3

Lesson 4: Optimizing multivariable functions (articles)

Maxima, minima, and saddle points

Google Classroom

Learn what local maxima/minima look like for multivariable function.

Background

What we're building to

Intuitively, when you're thinking in terms of graphs, local maxima of multivariable functions are peaks, just as they are with single variable functions.
The gradient of a multivariable function at a maximum point will be the zero vector, which corresponds to the graph having a flat tangent plane.
Formally speaking, a local maximum point is a point in the input space such that all other inputs in a small region near that point produce smaller values when pumped through the multivariable function $f$ ‍.

In general, the graph of a function with

n

variables exists in an

(n + 1)

-dimensional space, and its tangent is an

n

-dimensional space.

For example, the graph of a single variable function is two-dimensional and its tangent is a one-dimensional line; the graph of a two-variable function is three-dimensional and its tangent is a two-dimensional plane.

For functions with three variables or more, the tangent won't be a plane anymore, since a plane is necessarily two-dimensional.

The formal term for a subspace that has one dimension less than its ambient space is hyperplane. So, formally, the gradient of a multivariable function corresponds to a tangent hyperplane.

In this article, we will keep calling the tangent a plane, because all of our examples use two-variable functions. Nevertheless, it's important to remember that formally, we are talking about hyperplanes.

Optimizing in higher dimensions

One of the most important applications of calculus is its ability to sniff out the maximum or the minimum of a function.

Perhaps you find yourself running a company, and you've come up with some function to model how much money you can expect to make based on a number of parameters, such as employee salaries, cost of raw materials, etc., and you want to find the right combination of resources that will maximize your revenues.
Maybe you are designing a car, hoping to make it more aerodynamic, and you've come up with a function modelling the total wind resistance as a function of many parameters that define the shape of your car, and you want to find the shape that will minimize the total resistance.
In machine learning and artificial intelligence, the way a computer "learns" how to do something is commonly to minimize some "cost function" that the programmer has specified.

Local maxima and minima, visually

Let's start by thinking about those multivariable functions which we can graph: Those with a two-dimensional input, and a scalar output, like this:

f (x, y) = \cos (x) \cos (y) e^{- x^{2} - y^{2}}

I chose this function because it has lots of nice little bumps and peaks. We call one of these peaks a local maximum, and the plural is local maxima.

The point $(x_{0}, y_{0})$ ‍ underneath a peak in the input space (which in this case means the $x y$ ‍-plane) is called a local maximum point.
The output of a function at a local maximum point, which you can visualize as the height of the graph above that point, is the local maximum itself.

The word "local" is used to distinguish these from the global maximum of the function, which is the single greatest value that the function can achieve. If you are on the peak of a mountain, it's a local maximum, but unless that mountain is Mt. Everest, it is not a global peak.

I'll give you the formal definition of a local maximum point at the end of this article. Intuitively, it is a special point in the input space where taking a small step in any direction can only decrease the value of the function.

Similarly, if the graph has an inverted peak at a point, we say the function has a local minimum point at the value

(x, y)

above/below this point on the

x y

-plane, and the value of the function at this point is a local minimum. Intuitively, these are points where stepping in any direction can only increase the value of the function.

Stable points in one variable (review)

You may remember the idea of local maxima/minima from single-variable calculus, where you see many problems like this:

Concept check: For what value

x

is the function

f (x) = - (x - 2)^{2} + 5

the greatest? What is the maximum value?

x =

The maximum value of

f

In general, local maxima and minima of a function

f

are studied by looking for input values

a

where

f^{'} (a) = 0

. This is because as long as the function is continuous and differentiable, the tangent line at peaks and valleys will flatten out, in that it will have a slope of

0

Such a point

a

has various names:

Stable point
Critical point
Stationary point

All of these mean the same thing:

f^{'} (a) = 0

The requirement that

f

be continuous and differentiable is important, for if it was not continuous, a lone point of discontinuity could be a local maximum:

And if

f

is continuous but not differentiable, a local maximum could look like this:

In either case, talking about tangent lines at these maximum points doesn't really make sense, does it?

However, even when

f

is continuous and differentiable, it is not enough for the derivative to be

0

, since this also happens at inflection points:

This means finding stable points is a good way to start the search for a maximum, but it is not necessarily the end.

Stable points in two variables

The story is very similar for multivariable functions. When the function is continuous and differentiable, all the partial derivatives will be $0$ ‍ at a local maximum or minimum point.

\begin{aligned} \underset{Partial with respect to x}{\underset{⏟}{f_{x} (x_{0}, y_{0}, \dots)}} & = 0 \\ \underset{Partial with respect to y}{\underset{⏟}{f_{y} (x_{0}, y_{0}, \dots)}} & = 0 \\ ⋮ \end{aligned}

With respect to the graph of a function, this means its tangent plane will be flat at a local maximum or minimum. For instance, here is a graph with many local extrema and flat tangent planes on each one:

Khan Academy video wrapper

See video transcript

Saying that all the partial derivatives are zero at a point is the same as saying the gradient at that point is the zero vector:

\begin{aligned} \nabla f (x_{0}, y_{0}, \dots) \\ = [\begin{array}{c} f_{x} (x_{0}, y_{0}, \dots) \\ f_{y} (x_{0}, y_{0}, \dots) \\ ⋮ \end{array}] \\ = [\begin{array}{c} 0 \\ 0 \\ ⋮ \end{array}] \end{aligned}

People often write this more compactly like this:

\nabla f (x_{0}) = 0

The convention is that bold variable are vectors. So

x_{0}

is a vector of the input values

(x_{0}, y_{0}, \dots)

and

0

is the vector with all zeros.

Such an input

x_{0}

goes by the same various names as in the single-variable case:

Stable point
Stationary point
Critical point

The thinking behind the words "stable" and "stationary" is that when you move around slightly near this input, the value of the function doesn't change significantly. The word "critical" always seemed a bit over dramatic to me, as if the function is about to die near those points.

As with single variable functions, it is not enough for the gradient to be zero to ensure that a point is a local maximum or minimum. For one thing, you can still have something similar to an inflection point:

But there is also an entirely new possibility, unique to multivariable functions.

Saddle points

Consider the function

f (x, y) = x^{2} - y^{2}

. Let's make a few observations about what goes on around the origin

(0, 0)

Both partial derivatives are $0$ ‍ at this point:

\begin{aligned} \frac{\partial}{\partial x} (x^{2} - y^{2}) & = 2 x \to 2 (0) = 0 \\ \frac{\partial}{\partial y} (x^{2} - y^{2}) & = - 2 y \to - 2 (0) = 0 \end{aligned}

Therefore

(0, 0)

is a stable point.

When you just move in the $x$ ‍ direction around this point, the function looks like $f (x, 0) = x^{2} - 0^{2} = x^{2}$ ‍. The single-variable function $f (x) = x^{2}$ ‍ has a local minimum at $x = 0$ ‍.
When you just move in the $y$ ‍ direction around this point, meaning the function looks like $f (0, y) = 0^{2} - y^{2} = - y^{2}$ ‍. The single-variable function $f (y) = - y^{2}$ ‍ has a local maximum at $y = 0$ ‍.

In other words, the

x

and

y

directions disagree over whether this input should be a maximum or a minimum point. So even though

(0, 0)

is a stable point, and is not an inflection point, it cannot be a local maximum or local minimum!

Here's a video of this graph rotating in space:

Khan Academy video wrapper

See video transcript

Doesn't the region around

(0, 0, 0)

kind of have the shape of a horse's saddle?

Well, mathematicians thought so, and they had one of those rare moments of deciding on a good name for something: Saddle points. By definition, these are stable points where the function has a local maximum in one direction, but a local minimum in another direction.

Testing maximality/minimality

"Alright,"

I hear you saying,

"so it's not enough for the gradient to be

0

since you might have an inflection point or a saddle point. But how can you tell if a stable point is a local maximum or minimum?"

I'm glad you asked! This is the topic of the next article on the second partial derivative test. For now, let's finish things off with a formal definition of a local maximum.

Formal definition

I've said this before, but the reason to learn formal definitions, even when you already have an intuition, is to expose yourself to how intuitive mathematical ideas are captured precisely. It's good practice for thinking clearly, and it can also help to understand those times when intuition differs from reality.

In defining a local maximum, let's use vector notation for our input, writing it as

x

Formal definition of a local maximum: A scalar-valued function

f

has a local maximum at

x_{0}

if there exists some positive number

r > 0

, thought of as a radius, such that the following statement is true:

$f (x) \leq f (x_{0})$ ‍ for all $x$ ‍ such that $| | x - x_{0} | | < r$ ‍

That's a bit of a mouthful, so let's break it down:

Saying "

| | x - x_{0} | | < r

" means the variable

x

is within a distance

r

of the maximum point

x_{0}

. When

x

is two-dimensional this is the same as saying

x

lies inside a circle of radius

r

centered at the point

x_{0}

More generally, if

x

n

-dimensional, the set of all

x

such that

| | x - x_{0} | | < r

forms an

n

-dimensional ball with radius

r

centered at

x_{0}

We can then translate this definition from math-speak to something more closely resembling English as follows:

$x_{0}$ ‍ is a maximum point of $f$ ‍ if there is some small (ball-shaped) region in the input space around the point $x_{0}$ ‍ such that the highest possible value you can get for $f$ ‍ evaluated on points in that region is achieved at the point $x_{0}$ ‍.

Test your understanding: Write the formal definition for a local minimum, and think about what each component means as you write it down. (Resist the temptation to just copy down the words in the definition above.)

Summary

Intuitively, when you're thinking in terms of graphs, local maxima of multivariable functions are peaks, just as they are with single variable functions.
The gradient of a multivariable function at a maximum point will be the zero vector, which corresponds to the graph having a flat tangent plane.
Formally speaking, a local maximum point is a point in the input space such that all other inputs in a small region near that point produce smaller values when pumped through the multivariable function $f$ ‍.

Want to join the conversation?

Sort by:

Robert
Posted 8 years ago. Direct link to Robert's post “When reading this article...”
When reading this article I noticed the "Subject: Prometheus" button up at the top just to the right of the KA homesite link. Anyone else notice this?
Button navigates to signup pageButton navigates to signup page
(6 votes)
Answer
- Raymond Muller
  Posted 8 years ago. Direct link to Raymond Muller's post “Nope. I have a "Subject:...”
  Nope. I have a "Subject: Multivariable Calculus" button. Glitch?
  Comment on Raymond Muller's post “Nope. I have a "Subject:...”
  (4 votes)
Mohammed Ghaïth
Posted a year ago. Direct link to Mohammed Ghaïth's post “Saddle point and inflecti...”
Saddle point and inflection point are two things I have a hard time differentiating! They are not the same, graphically, but what are the key mathematical differences between them both?
Button navigates to signup pageButton navigates to signup page
(2 votes)
Answer
- Venkata
  Posted a year ago. Direct link to Venkata's post “Saddle points are used fo...”
  Saddle points are used for multivariable functions, while inflection points are found in the case of single variable functions. So, that's one point of difference.
  
  As for what they are, saddle points are extremum points which are neither local minima or maxima (How are they extrema then? Well, they show up when you try to find extrema, but once you use the second derivative test on them, they return 0)
  
  Inflection points are points where functions change concavity (go from increasing to decreasing or vice versa).
  Comment on Venkata's post “Saddle points are used fo...”
  (5 votes)
Andrea Menozzi
Posted 6 years ago. Direct link to Andrea Menozzi's post “what R should be? any val...”
what R should be? any value? does the limit of R tends to zero?
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- Sam Tan
  Posted 4 years ago. Direct link to Sam Tan's post “The specific value of r i...”
  The specific value of r is situational, depending on how "local" you want your max/min to be.
  Button navigates to signup page
  (2 votes)
Andrea Menozzi
Posted 6 years ago. Direct link to Andrea Menozzi's post “f(x)≥f(x0) why it is allo...”
f(x)≥f(x0) why it is allowed to be greater or EQUAL ? isn't it just greater?
Button navigates to signup pageButton navigates to signup page
(3 votes)
Answer
- Alex Sloan
  Posted 4 years ago. Direct link to Alex Sloan's post “Well think about what hap...”
  Well think about what happens if we do what you are suggesting. We cant have the point x = x0 then yet when we say for all x we mean for the entire domain of the function. x0 thus must be part of the domain if we are able to evaluate it in the function. So what happens when x does equal x0? If the definition was just > and not >= then we would find that the condition is not true and thus the point x0 would not be a maximum which is not what we want.
  Button navigates to signup page
  (2 votes)
Will Simon
Posted 6 years ago. Direct link to Will Simon's post “It is inaccurate to say t...”
It is inaccurate to say that "this [the derivative being 0] also happens at inflection points." The function f(x)=sin(x) has an inflection point at x=0, but the derivative is not 0 there. I think what you mean to say is simply that a function's derivative can equal 0 at a point without having an extremum at that point, which is related to the fact that the second derivative at that point is 0, i.e. neither positive nor negative (i.e. the point is an inflection point). Maybe you meant that "this also can happen at inflection points."
Button navigates to signup pageButton navigates to signup page
(2 votes)
Answer
shivnaren
Posted 3 years ago. Direct link to shivnaren's post “_In machine learning and ...”
In machine learning and artificial intelligence, the way a computer "learns" how to do something is commonly to minimize some "cost function" that the programmer has specified.

I think it should be loss function--not cost function?
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer
- Jerry Nilsson
  Posted 3 years ago. Direct link to Jerry Nilsson's post “Well, if doing A costs B,...”
  Well, if doing A costs B, then by doing A you lose B.
  So, whether you call it cost function or loss function is just semantics – the meaning is the same.
  Comment on Jerry Nilsson's post “Well, if doing A costs B,...”
  (3 votes)
zk306950
Posted 4 years ago. Direct link to zk306950's post “Is the following true whe...”
Is the following true when identifying if a critical point is an inflection point? A critical point of function F (the gradient of F is the 0 vector at this point) is an inflection point if both the F_xx (partial of F with respect to x twice)=0 and F_yy (partial of F with respect to y twice)=0 and of course the Hessian must be >0 to avoid being a saddle point or inconclusive.
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer
thd
Posted 4 months ago. Direct link to thd's post “A purely mathematical que...”
A purely mathematical question:In the definition of local minima/maxima shouldn't there be a 'if and only if' instead of just 'if'?
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer
sprincejindal
Posted 2 years ago. Direct link to sprincejindal's post “When talking about Saddle...”
When talking about Saddle point in this article. It says 'The single-variable function f(x) = x^2 has a local minimum at x=0, and
The single-variable function f(y) = -y^2 has a local maximum at y = 0.

Can someone please explain why x is minimum and y is maximum and not vice versa?
Button navigates to signup pageComment on sprincejindal's post “When talking about Saddle...”
(1 vote)
Answer
- Mohammed Ghaïth
  Posted a year ago. Direct link to Mohammed Ghaïth's post “Imagine the respect graph...”
  Imagine the respect graphs of the two, and observe the concavity.
  Button navigates to signup page
  (1 vote)
kashmalahassan015
Posted 8 years ago. Direct link to kashmalahassan015's post “questions of triple deriv...”
questions of triple derivative
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer