Main content

## Lagrange multipliers and constrained optimization

Current time:0:00Total duration:12:28

# The Lagrangian

## Video transcript

- [Lecturer] All right, so
today I'm gonna be talking about the Lagrangian. Now we talked about Lagrange multipliers. This is a highly related concept. In fact, it's not really
teaching anything new. This is just repackaging
stuff that we already know. So to remind you of the setup, this is gonna be a constrained
optimization problem setup so we'll have some kind
of multivariable function. F of x, y, and the one
I have pictured here is, let's see, it's x squared times e to the y times y so what I have shown here is a contour line for this function. So that is, we say what
happens if we set this equal to some constant, and we ask
about all values of x and y such that this holds, such that this function
outputs that constant, and if I choose a different constant, and that contour line can
look a little bit different, it's kinda nice that it has similar shapes so that's the function, and
we're trying to maximize it. The goal is to maximize this guy, and of course, it's not just that. The reason we call it a
constrained optimization problem is 'cause there's some kind of constraint, some kind of other function, g of x, y. In this case, x squared plus y squared, and we want to say that this has to equal some specific amount. In this case, I'm gonna
set it equal to four. So we say you can't look at any x, y to maximize this function. You are limited to the values of x and y that satisfy this property, and I talked about this in
the last couple of videos, and kind of the cool thing
that we found was that you look through the various
different contour lines of f, and the maximum will be
achieved when that contour line is just perfectly parallel
to this contour of g, and you know, a pretty classic example for what these sorts of things could mean, or how it's used in
practice is if this was, say a revenue function
for some kind of company. You're kind of modeling your revenues based on different choices you could make running that company, and the constraint that
you'd have would be, let's say a budget so
I'm just gonna go ahead and write budget or B for budget here so you're trying to maximize revenues, and then you have some
sort of dollar limit for what you're willing to spend, and these, of course, are just
kind of made-up functions. You'd never have a budget
that looks like a circle and this kind of random
configuration for your revenue but in principle, you
know what I mean, right? So the way that we took advantage
of this tangency property, and I think this is pretty clever. Let me just kind of redraw it over here. You're looking at the point
where the two functions are just tangent to each
other is that the gradient, the gradient vector for
the thing we're maximizing, which in this case is
R, is gonna be parallel or proportional to the gradient
vector of the constraint, which in this case is B,
is gonna be proportional to the gradient of the constraint, and what this means is that
if we were going to solve a set of equations, what you set up is you compute that gradient of R, and it'll involve two
different partial derivatives, and you set it equal
not to the gradient of B 'cause it's not necessarily equal to the gradient of B but it's proportional with some kind of
proportionality constant, lambda. That's kind of a squarely lambda. Lambda. That one doesn't look
good either, does it? Why are lambdas so hard to draw? Right, that looks fine. So the gradient of the revenue is proportional to the
gradient of the budget, and we did a couple of examples of solving this kind of thing. This gives you two separate equations from the two partial derivatives, and then you use this right
here, this budget constraint as your third equation,
and the Lagrangian, the point of this video,
this Lagrangian function is basically just a way to
package up this equation along with this equation
into a single entity so it's not really adding new information, and if you're solving things by hand, it doesn't really do anything for you but what makes it nice is
that it's something easier to hand a computer, and
I'll show you what I mean. So I'm gonna define the Lagrangian itself, which we write with this kind
of funky looking script, L, and it's a function with the same inputs that your revenue function or the thing that you're
maximizing has along with lambda, along with that Lagrange multiplier, and the way that we define it, and I'm gonna need some extra room so I'm gonna say it's equal to, and kinda define it down here, the revenue function or whatever it is that you're maximizing, the
function that you're maximizing minus lambda, that Lagrange multiplier so that's just another
input to this new function that we're defining multiplied
by the constraint function, in this case, B evaluated at x, y minus whatever that constraint value is. In this case, I put in four
so you write minus four. If we wanted to be more general, maybe we would write b for
whatever your budget is so over here, you're
subtracting off little b so this here is a new
multivariable function, right? It's something where you
could input x, y, and lambda, and just kind of plug it all in, and you'd get some kind of value, and remember b, in this
case, is a constant so I'll go ahead and write that that this right here is
not considered a variable. This is just some constant. Your variables are x, y, and lambda. And this would seem like a
totally weird and random thing to do if you just saw it out of context, or if it was unmotivated
but what's kind of neat, and we'll go ahead and
work through this right now is that when you take this, is that when you take the
gradient of this function called the Lagrangian, and
you set it equal to zero, that's gonna encapsulate all
three equations that you need, and I'll show you what I mean by that. So let's just remember the gradient of L, that's a vector. That's got three different components since L has three different inputs. You're gonna have the
partial derivative of L with respect to x. You're gonna have the
partial derivative of L with respect to y. And then finally the
partial derivative of L with respect to lambda,
our Lagrange multiplier, which we're considering
an input to this function. And remember whenever we write
that the vector equals zero, really we mean the zero vector. Often you'll see it in
bold if it's in a textbook but what we're really saying is we set those three different functions, the three different partial
derivatives all equal to zero so this is just a nice like
closed form, compact way of saying that all of
its partial derivatives is equal to zero, and let's go ahead and think about what those
partial derivatives actually are. So this first one, the
partial with respect to x, partial derivative of the
Lagrangian with respect to x. It's kinda funny. Now you have all these curly symbols, the curly d, the curly l. It makes it look like you're
doing some truly advanced math but really it's just kind of
artificial fanciness, right? But anyway so we take
the partial derivative with respect to x, and
what that equals is well, it's whatever the partial derivative of R with respect to x is
minus, and then lambda. From x's perspective, lambda
just looks like a constant. So this can be lambda. And then this inside the parenthesis, the partial derivative of
that with respect to x, well, it's gonna be whatever
the partial derivative of B is with respect to x but
subtracting off that constant, that doesn't change the derivative so this right here is the
partial derivative of lambda with respect to x. Now if you set that equal to zero. I don't know. I've kind of run out of
room on the right here but if you set that equal to zero, that's the same as just saying
that the partial derivative of R with respect to x equals lambda times the partial derivative
of B with respect to x, and if you think what's gonna happen when you unfold this property, the gradient of R is
proportional to the gradient of B written up here. That's just the first
portion of this, right? If we're setting the gradients equal, then the first component of that is to say that the partial derivative of R with respect to x is equal to lambda times the partial derivative
of B with respect to x. And then if you do this for y, if we take the partial derivative of this Lagrangian function with respect to y, it's very similar, right? It's gonna be, well, you just
take the partial derivative of R with respect to y. In fact, it all looks just identical. Whatever R is, you take
as partial derivative with respect to y, and
then we subtract off. Lambda looks like a constant
as far as y is concerned, and then that's multiplied by, well, what's the partial derivative of this term inside the parenthesis with respect to y? Well, it's the partial
of B with respect to y. And again, again, if you imagine
setting that equal to zero, that's gonna be the same as setting this partial derivative term equal to lambda times this
partial derivative term, right? You kind of just bring
one to the other side. So this second component of our Lagrangian equals zero equation is just the second
function that we've seen in a lot of these examples
that we've been doing where you set one of the gradient vectors proportional to the other one, and the only real difference here from stuff that we've seen already, and even then it's not that different is that what happens when we
take the partial derivative of this Lagrangian with respect to lambda, and I'll go ahead and give it that kind of green lambda color here. Well, when we take that
partial derivative, if we kinda look up with the
definition of the function, R, R never has a lambda in it, right? It's purely a function of x and y so that looks just like a constant when we're differentiating
with respect to lambda so that's just gonna be zero when we take its partial derivative. And then this next
component, B of x, y minus b, all of that just looks like a constant as far as lambda is concerned, right? There's x's, there's y,
there is this constant b but none of these things
have lambdas in them so when we take the partial derivative with respect to lambda, this just looks like some big
constant times lambda itself. So what we're gonna get is I guess we're subtracting off, right? Up here, we're kind of
writing a minus sign. We're subtracting off all the stuff that was
in those parenthesis, B of x, y minus b, that constant. And this whole thing, if we set that whole thing equal to zero, well, that's pretty much the same as setting B of x, y
minus b equal to zero, and that, that's really
just the same as saying, "Hey, we're setting B of x, y "equal to that little b," right? Setting this partial
derivative of the Lagrangian with respect to the Lagrange
multiplier equal to zero boils down to the constraint, right? The third equation that we need to solve. So in that way, setting the gradient of this Lagrangian function equal to zero is just a very compact way of packaging three separate
equations that we need to solve the constrained
optimization problem, and I'll emphasize that in practice, if you actually see a function for R for the thing that you're maximizing, and a function for the budget,
it's much better I think to just directly think about
these parallel gradients, and kind of solve it from there because if you construct the Lagrangian, and then compute its gradient, all you're really doing
is repackaging it up only to unpackage it again but the point of this, kind of the reason that this is a very useful construct is that computers often
have really fast ways of solving things like this, things like the gradient of
some function equals zero, and the reason is because
that's how you solve unconstrained maximization
problems, right? This is very similar
to as if we just looked at this function L out of
context, and we're asked, "Hey, what is its maximum value? "What are the critical
points that it has?" And you said it's gradient equal to zero. So kind of the whole
point of this Lagrangian is that it turns our
constrained optimization problem involving R and B and this
new made-up variable lambda into an unconstrained optimization problem where we're just setting the
gradient of some function equal to zero so computers can
often do that really quickly so if you just hand the
computer this function, it will be able to find you an answer whereas it's harder to say, "Hey computer, "I want you to think about when
the gradients are parallel, "and also consider this
constraint function." It's just sort of a cleaner
way to package it all up. So with that, I'll see you next video where I'm gonna talk
about the significance of this lambda term, how it's
not just a ghost variable but it actually has a
pretty nice interpretation for a given constrained problem.