Quadratic approximation

Background:

• Local linearization
• Graphs
• Second partial derivatives

What we're building to

• $f$f is a function with multi-dimensional input and a scalar output.
• $\nabla f(\textbf{x}_0)$del, f, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis is the gradient of $f$f evaluated at $\textbf{x}_0$start bold text, x, end bold text, start subscript, 0, end subscript.
• $\textbf{H}_f(\textbf{x}_0)$start bold text, H, end bold text, start subscript, f, end subscript, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis is the Hessian matrix of $f$f evaluated at $\textbf{x}_0$start bold text, x, end bold text, start subscript, 0, end subscript.
• The vector $\textbf{x}_0$start bold text, x, end bold text, start subscript, 0, end subscript is a specific input, the one we are approximating near.
• The vector $\textbf{x}$start bold text, x, end bold text represents the variable input.
• The approximation function, $Q_f$Q, start subscript, f, end subscript, has the same value as $f$f at the point $\textbf{x}_0$start bold text, x, end bold text, start subscript, 0, end subscript, all its partial derivatives have the same value as those of $f$f at this point, and all its second partial derivatives have the same value as those of $f$f at this point.

Tighter and tighter approximations

Zero-order approximation

First-order approximation

Second-order approximation.

"Quadratic" means product of two variables

Graphs of quadratic functions

Reminder on the local linearization recipe

• Start with the constant term $f(x_0, y_0)$f, left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis, so that our approximation at least matches $f$f at the point $(x_0, y_0)$left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis.
• Add on linear terms $\blueE{f_x(x_0, y_0)}(x - x_0)$start color #0c7f99, f, start subscript, x, end subscript, left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis, end color #0c7f99, left parenthesis, x, minus, x, start subscript, 0, end subscript, right parenthesis and $\redE{f_y(x_0, y_0)}(y - y_0)$start color #bc2612, f, start subscript, y, end subscript, left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis, end color #bc2612, left parenthesis, y, minus, y, start subscript, 0, end subscript, right parenthesis.
• Use the constants $\blueE{f_x(x_0, y_0)}$start color #0c7f99, f, start subscript, x, end subscript, left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis, end color #0c7f99 and $\redE{f_y(x_0, y_0)}$start color #bc2612, f, start subscript, y, end subscript, left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis, end color #bc2612 to ensure that our approximation has the same partial derivatives as $f$f at the point $(x_0, y_0)$left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis.
• Use the terms $(x - x_0)$left parenthesis, x, minus, x, start subscript, 0, end subscript, right parenthesis and $(y - y_0)$left parenthesis, y, minus, y, start subscript, 0, end subscript, right parenthesis instead of simply $x$x and $y$y so that we don't mess up the fact that our approximation equals $f(x_0, y_0)$f, left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis at the point $(x_0, y_0)$left parenthesis, x, start subscript, 0, end subscript, comma, y, start subscript, 0, end subscript, right parenthesis.

Finding the quadratic approximation

• Try it! Take the second partial derivative with respect to $x$x of every term in the expression of $Q_f(x, y)$Q, start subscript, f, end subscript, left parenthesis, x, comma, y, right parenthesis above, and notice that they all go to zero except for the $\blueE{a}(x - x_0)^2$start color #0c7f99, a, end color #0c7f99, left parenthesis, x, minus, x, start subscript, 0, end subscript, right parenthesis, squared term.

Example: Approximating $\sin(x)\cos(y)$sine, left parenthesis, x, right parenthesis, cosine, left parenthesis, y, right parenthesis

$f\left(\dfrac{\pi}{3}, \dfrac{\pi}{6}\right) =$f, left parenthesis, start fraction, pi, divided by, 3, end fraction, comma, start fraction, pi, divided by, 6, end fraction, right parenthesis, equals

Check

[Answer]

$f_x(x, y) =$f, start subscript, x, end subscript, left parenthesis, x, comma, y, right parenthesis, equals

$f_x\left(\dfrac{\pi}{3}, \dfrac{\pi}{6}\right) =$f, start subscript, x, end subscript, left parenthesis, start fraction, pi, divided by, 3, end fraction, comma, start fraction, pi, divided by, 6, end fraction, right parenthesis, equals

Check

[Answer]

$f_y(x, y) =$f, start subscript, y, end subscript, left parenthesis, x, comma, y, right parenthesis, equals

$f_y\left(\dfrac{\pi}{3}, \dfrac{\pi}{6}\right) =$f, start subscript, y, end subscript, left parenthesis, start fraction, pi, divided by, 3, end fraction, comma, start fraction, pi, divided by, 6, end fraction, right parenthesis, equals

Check

[Answer]

$f_{xx}(x, y) =$f, start subscript, x, x, end subscript, left parenthesis, x, comma, y, right parenthesis, equals

$f_{xx}\left(\dfrac{\pi}{3}, \dfrac{\pi}{6}\right) =$f, start subscript, x, x, end subscript, left parenthesis, start fraction, pi, divided by, 3, end fraction, comma, start fraction, pi, divided by, 6, end fraction, right parenthesis, equals

Check

[Answer]

$f_{xy}(x, y) =$f, start subscript, x, y, end subscript, left parenthesis, x, comma, y, right parenthesis, equals

$f_{xy}\left(\dfrac{\pi}{3}, \dfrac{\pi}{6}\right) =$f, start subscript, x, y, end subscript, left parenthesis, start fraction, pi, divided by, 3, end fraction, comma, start fraction, pi, divided by, 6, end fraction, right parenthesis, equals

Check

[Answer]

$f_{yy}(x, y) =$f, start subscript, y, y, end subscript, left parenthesis, x, comma, y, right parenthesis, equals

$f_{yy}\left(\dfrac{\pi}{3}, \dfrac{\pi}{6}\right) =$f, start subscript, y, y, end subscript, left parenthesis, start fraction, pi, divided by, 3, end fraction, comma, start fraction, pi, divided by, 6, end fraction, right parenthesis, equals

Check

[Answer]

Vector notation using the Hessian

• The boldfaced $\textbf{x}$start bold text, x, end bold text represents the input variable(s) as a vector,
  $\begin{aligned} \quad \textbf{x} &= \left[ \begin{array}{c} x \\ y \\ \vdots \end{array} \right] \end{aligned}$
  Moreover, $\textbf{x}_0$start bold text, x, end bold text, start subscript, 0, end subscript is a particular vector in the input space. If this has two components, this formula for $Q_f$Q, start subscript, f, end subscript is just a different way to write the one we derived before, but it could also represent a vector with any other dimension.
• The dot product $\nabla f(\textbf{x}_0) \cdot (\textbf{x} - \textbf{x}_0)$del, f, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, dot, left parenthesis, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis will expand into the sum of all terms of the form $f_{x}(\textbf{x}_0)(x - x_0)$f, start subscript, x, end subscript, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, left parenthesis, x, minus, x, start subscript, 0, end subscript, right parenthesis, $f_{y}(\textbf{x}_0)(y - y_0)$f, start subscript, y, end subscript, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, left parenthesis, y, minus, y, start subscript, 0, end subscript, right parenthesis, etc. if this is not familiar from the vector notation for local linearization, work it out for yourself in the case of $2$2-dimensions to see!
  [Solution]
• The little superscript $T$T in the expression $(\textbf{x} - \textbf{x}_0)^\mathrm{T}$left parenthesis, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, start superscript, T, end superscript indicates "transpose". This means you take the initial vector $(\textbf{x} - \textbf{x}_0)$left parenthesis, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, which looks something like this:

  $(\textbf{x} - \textbf{x}_0) = \left[ \begin{array}{c} x - x_0 \\ y - y_0 \end{array} \right]$

  Then you flip it, to get something like this:

  $(\textbf{x} - \textbf{x}_0)^{\mathrm{T}} = \left[ \begin{array}{c} x - x_0 & & y - y_0 \end{array} \right]$

• $\textbf{H}_f(\textbf{x}_0)$start bold text, H, end bold text, start subscript, f, end subscript, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis is the Hessian of $f$f.
• The expression $(\textbf{x} - \textbf{x}_0)^\mathrm{T} \textbf{H}_f(\textbf{x}_0)(\textbf{x} - \textbf{x}_0)$left parenthesis, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, start superscript, T, end superscript, start bold text, H, end bold text, start subscript, f, end subscript, left parenthesis, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis, left parenthesis, start bold text, x, end bold text, minus, start bold text, x, end bold text, start subscript, 0, end subscript, right parenthesis might seem complicated if you have never come across something like it before. This way of expressing quadratic terms is actually quite common in vector-calculus and vector-algebra, so it's worth expanding an expression like this at least a few times in your life. For example, try working it out in the case where $\textbf{x}$start bold text, x, end bold text is two-dimensional to see what it looks like.
  [Solution]
  You should find that it is exactly $2$2 times the quadratic portion of the non-vectorized formula we derived above.

What's the point?

• Computation: Even if you never have to write out a quadratic approximation, you may one day need to program a computer to do it for a particular function. Or even if you are relying on someone else's program, you may need to analyze how and why the approximation is failing in some circumstance.
• Theory: Being able to reference a second-order approximation helps us to reason about the behavior of general functions near a point. This will be useful later in figuring out if a point is a local maximum or minimum.