If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

Main content

# The Gram-Schmidt process

## Video transcript

let's say I have a set of linearly independent vectors v1 v2 all the way to VK that are a basis a basis for V we've seen this many times before that's nice it's a basis but we've learned over the last few videos it would be even better to have an orthonormal basis to have a an ortho normal basis for V orthonormal basis so what we're going to address in this video is is there some way that we can construct an orthonormal basis for V given that we already have this basis which I'm assuming isn't orthonormal but it'll work whether or not it's orthonormal and it'll just kind of generate another orthonormal basis but can we somehow just given any basis generate an orthonormal basis for V and then be able to benefit from all of the properties of having an orthonormal basis so let's see if we can if we can make some headway here so let's let's think about it I guess for simple cases let's say that I just have a one-dimensional subspace let's call that V one for it's a one-dimensional subspace and I'm just going to say it's the span it's the span of just this vector of the span of just the vector V one now or you could say that V one is the basis for or the vector V 1 is a basis for the subspace V 1 now if I had this this simple case how can I ensure that this is orthonormal well what I could do is I could define some vector let's call it U 1 let's call it u 1 that's essentially equal to you know obviously this is orthogonal to all of the other guys there aren't any other guys here argue the zero vectors there but there aren't any other members of this set so it's orthogonal to everything else because there isn't anything else and then to make its length equal to 1 you can just take this vector and divide it by its length so if we take if we define some vector u 1 to be equal to V 1 divided by the length of the length of V 1 then the length of U 1 the length of you one is going to be what it's going to be the length of v1 length of v1 over let me write it this way over the length of v1 like this this is just a constant right here so this is just going to be 1 over the length of v1 that's just you know that could be like five times the length of v1 which is just equal to one so this one will have a length of one so just like that if we define if we say that if we have the set of just u1 we could say that just a set of u1 is an ortho ortho normal basis basis for v1 for the subspace v1 now that was trivially easy I mean if if my if K happens to be one we're done that was you know super easy we just have to divide by your length you just have to essentially just normalize this vector and you have an orthonormal basis because there's nothing else to be orthogonal with but let's complicate the issue a little bit let's say we go to two dimensions let's say we have a subspace V two that's the span of let's say it's the first two of these vectors it's the span of v1 it's the span of v1 and the vector and the vector v2 now we know that v1 v1 can easily be represented as or something that v1 is a linear combination of u1 how do I know that well I could just multiply both sides of this times the length of V 1 we have u1 times the length of v1 is equal to v1 so we could say that this is the same thing this is equivalent to saying the span the span of the vector u1 and the vector v2 where you want is the vector we got up here how can I say this because anything that can be linear combination of these guys can be linear combination of those guys because wherever you had a v1 you can replace the v1 with the linear combination of u1 that gives you v1 so you just multiply u1 times that scalar and you get I think you you get the point but how can we ensure that this an orthonormal set so what do we do let's graph it so this is going to be a plane in RN so let's just make our chalkboard here or a video board here the plane so we have u1 which is a unit vector it has length 1 so that is u 1 and u 1 at well V 1 and V 2 are linearly independent that's by definition of a basis so you can't represent V 2 as a linear as a linear multiple or a linear combination of V 1 and likewise you can't represent V 2 as a linear combination of U 1 because Hugh hunt is a linear combination of U on so V 1 is not going to be on the line spanned by u1 so I could actually draw that the line spanned by u1 is just like this it's this is the line spanned by u1 let me don't look better than that I want to know I make it too dark so this is the line and do one last try the line spanned by u1 looks like this it looks like that it's just a line and that is the subspace that is the subspace v1 right with a span of u1 so this is equal to the span of u1 all we did is normalize v1 right there to get to u1 so the span of v1 is the same as span of u1 that is this subspace that's this line in RN now we have vector v2 which is linearly independent from v1 and u1 so v2 would look you know let's just say it looks something like that v2 now what we want to do is replace v2 with another vector that is definitely orthogonal to this and we'll still be able to construct v2 with some combination of this and our new vector well the most obvious vector would be some vector that is orthogonal that is orthogonal to V 1 so it's orthogonal sorts or it's a member of the orthogonal complement of V 1 so it's a member of the orthogonal complement of V 1 and if I so if you look at it just visually here if I add some member of V 1 to that member of the orthogonal complement of V 1 I'm going to get V 2 in fact we've seen that multiple times you know this guy right here let's say that we know that any vector in RN let's say V 2 can be represented as a sum of two vectors I'll call them X and Y where X is a member X is a member of V one and Y is a member of the orthogonal complement of V one we've seen that multiple times now what was this thing by definition what was this thing by definition we're looking for this thing this is the X so that is the X and then this is the Y right that is the Y we're looking for the Y because if we can find if we can find this vector Y then if we replace if we place V with that vector Y we'll still be able to generate V because you can take a multiple of U and add it to Y and get V and so anything that you were able to generate with V before v2 before you cannot generate with this with our with our u1 multiples of that plus multiples of our new vector so we're trying to solve we're trying to figure out what this vector Y is right there so how do we do it well it's going to be v2 minus this vector X right and what is this vector X by definition this vector X is by definition the projection of v2 it's the projection of v2 onto the subspace and v1 so are the vector that we're trying to find the vector Y that if we find it and that we can replace v2 with y the vector Y is just equal to v2 it's just equal to v2 let me write it this way minus the projection of v2 onto v1 that's what Y is and remember if we can replace v2 the reason why the span of u1 v2 is the same thing as the span is the span as the span of u1 let me call this y2 here we just call it Y 2 because we're probably gonna have to do use wise in the future you want let me why to so the reason why this span is the same thing as the span of u1 and why to is because I can generate v2 with a linear combination of u1 and y2 right I can scale up u1 and then add y2 and I'll get to v2 so anything that I could have generated with v2 I can get with a linear combination of these guys that's why these are equivalent and what's neat about this now is that these guys orthogonal complement to B or these guys orthogonal relative to each other right by definition Y was a member of the orthogonal complement so if you dot these two guys you're going to get zero and how can we actually solve for it well this is it this is useful as well because V 1 V 1 has an orthonormal basis right B one has an orthonormal basis has ortho I'll just write ortho basis I'm tired of writing orthonormal and we saw I think it was two or three videos ago that there's a vet the neat thing about orthonormal basis is it's very easy to determine the projection onto those basis it's essentially let me write it over here the projection the projection of the vector v2 onto the subspace v1 is equal to it's equal to v2 let me write it this way v2 dotted with the subspaces V ones first basis vector which is the vector sorry which is the vector u1 right that's its first basis vector we're dealing with an orthonormal basis times u1 and then if we had more basis vectors we would say plus v2 dotted with our next basis vector times that base basis vector so on and so forth but v1 only has one basis vector it only has the basis vector u1 right here right it was only spanned by that so we can rewrite this thing right here y2 y2 this vector right here that I'm going to replace v2 with is equal to v2 is equal to v2 minus the projection of v2 onto the subspace v1 onto that line which is just this v2 dot u1 times the vector you won and just like that we've solved for y so we have a a basis for v2 where this guy and this guy are orthogonal this guy is a unit vector he's been normalized but this guy hasn't been normalized yet so to normalize it let's just divide let's just define some other vector as u2 and do the same thing let's just normalize it so u2 is equal to Y 2 divided by the length of Y 2 divided by the length of Y 2 and so and that's so now we can say that the subspace V 2 is equal to the span the span of u1 u 1 and instead of Y 2 I'll put you two there because I can generate Y 2 by just scaling up YouTube so u 2 and what's neat about this now is I have two unit vectors or two normalized vectors and they're orthogonal with respect to each other and they span with V 1 and V 2 spanned originally now we need to keep going what about if we want to go to you know we have a V 3 here what do we do what we do the same idea we do the same idea so let's define the subspace V 3 this is going to be a three dimensional subspace once we get beyond V 3 it becomes a little hard to visualize but I think you're going to see the pattern after this step if we define V 3 is equal to the span the span of these guys u 1 u 2 and then in our original basis V 3 I didn't write it there but there's a V 3 there and V 3 so in our original non orthonormal basis we have V 3 now what is what is this going to look like and or how can we turn this into an orthonormal basis so if you if you visualize all of these so the span of u1 and u2 vr subspace V 2 is just going to be a plane it's just going to be a plane in r3 it's going to look like that right there and so we have our new span is going to be everything in that plane all linear combinations of things in that plane plus linear combinations of our vector V 3 which is linearly independent from these guys because it was linear independent from the stuff we use to construct these guys so v3 is going to pop out of this plane it can't be represented by a linear combination of those guys so let's say v3 pops out of the plane v3 like that now we want another vector that can represent everything of this of this span but it is orthogonal to these dudes right here or if you another way to think about its orthogonal to the plane so we want another vector that is orthogonal to the plane so let's call that vector Y y3 r i sub 3 not Y to the third power y sub 3 and if we figure out our Y sub 3 we can replace we can replace that we can replace v3 with it because v3 can be represented as a linear combination of U 1 and u 2 right that's going to be a linear combination of U 1 u 2 is going to be some vector in this plane plus y 3 I can represent this guy with this green vector plus this vector right here so if we replace them with Y 3 we can still get to V 3 and we can still get to all the linear combinations that V 3 can help construct so what is this Y 3 well by the same exact logic this green vector right here is the projection the projection of my vector V 3 onto the subspace V 2 onto the subspace V 2 and your Y 3 in your what your vector Y 3 is just equal to the vector V 3 V 3 minus the projection of v3 onto V 2 onto V 2 so what is what is that going to be well the projection I'll write it here the projection onto the subspace V 2 of V 3 of the vector V 3 is going to be equal to we're going to use the same exact logic we did before we see we saw this two or three videos ago because V 2 is defined with as with an orthonormal basis we can say that the projection of v3 onto that subspace is V 3 dot our first basis vector dot u 1 times our first basis vector plus V 3 dot our second basis vector are a second orthonormal basis vector times our second orthonormal basis vector it's that easy that's one of the neat things about having an orthonormal basis was this is an orthonormal basis so we can define the projection in this way so Y is just going to be v3 minus this business right here and I'll just write it out just so Y 3 is going to be V 3 V 3 minus the projection of v3 onto V 2 so minus minus that business right there let me copy it and then let me paste it and then you get Y 3 and y 3 is nice so if we replace this with Y 3 which we can because we can now get V 3 as a linear combination of these guys and Y 3 that's nice because all of these guys are orthogonal with respect to each other but Y 3 is not it does not have length 1 yet it hasn't been normalized so we can just replace Y 3 with another unit vector we'll just say u 3 is equal to Y 3 divided by the length of Y 3 whatever that might be but we can if we have what Y 3 is we can figure this length divided by that and then if you put so this is equal to the span of u1 u2 and u3 and u 3 so u 3 by might be a scaled down version of y3 but you could just you know we're talk about linear combinations when we talk about span so you can scale them up and add it to the projection you're still going to get V 3 and now these are all normalized vectors and so you would now have an orthonormal basis for the for the subspace V 3 I think you get the pattern now you can keep doing this now you want your if you're curious about you can define V 4 you know I we and when I define the problem I just said that we're dealing with you know a K dimensional subspace we go up to VK but you just keep going until you go to your K if K was 3 we would be done if K is 4 if K is 4 you say you define a subspace V 4 is equal to the span of u1 u2 you three and then throw our non orthonormal vector or v4 into the mix and so you can replace v4 you can replace v4 with y4 is equal to v4 minus the projection becomes a little hard to visualize you're not projecting onto a three-dimensional space the projection of V four on to our onto our subspace V three completely analogous to this it's just v3 is now a three dimensional space not a plane you're finding its projection onto this this is definitely going to be orthogonal to everything else in to our subspace V three and you can construct v4 with y4 because V 4 V 4 by definition V 4 is equal to we can just rearrange this y4 plus the projection of V 4 onto v3 so you can construct V 4 with y4 and some linear combination of these guys so I can replace this guy with y4 and then I would normalize y4 I would divide it by its length and I get you four and I just keep doing that until you know I got up to K you know if I do the file I do the process over and over and over again and this process of creating an orthonormal of an orthonormal basis is called this is called the gram-gram Schmidt Schmidt process and might seem a little abstract the way I did it here but in the next video I'm actually going to find orthonormal basis for subspaces you'll see it's not too bad when you actually deal with real numbers