I am puzzled. Is this not an example of computing in a unnecessarily complicated way? If we pick the order of vectors to be ortonormalized in a smarter(?) way :first and third are orthogonal so they just need to be normalized with (2)^(-1/2).....In the last ortogonalisation Sal realises that the first and last vectors are perpendicular, but does not comment that the whole process could/should have been done in a other order

You are correct, that this could be made simpler by recognizing the first and last vectors are orthogonal, and thus you can just normalize them to have two orthonormal basis vectors out of 3 right off the bat. However, the purpose of the video is to show the Graham Schmidt process from beginning to end with 3 basis vectors which can be applied to ANY set of basis vectors, not just use a trick available in this special case. The result for this example is some unnecessary computation, but this is sacrificed to provide a through and through example that can be applied to any set of 3 basis vectors. Note also that in more complex situations, it may not be immediately obvious that two/more vectors in your basis are orthogonal, so applying the Graham Schmidt processes is a rote but guaranteed way of generating an orthonormal basis.

Hello Khanacademy, I understood the concept explained. I tried using the gram-schmidt method on vectors V1,V2 and V3, which have complex random variables. After the process the vectors were not orthogonal to each other. But when I tried it with real numbers it worked. Please I need more information on this. Thank you.

Some of these things don't work with complex numbers, all the stuff Sal is explaining is for R^n, not C^n.

This feels like a really dumb question...I do not understand how the projection of V2 onto V1 equals (V2 times U1) times U1. I have seen all of the videos preceding but I just can't get it. Could someone explain this to me please?

v2 dot u1 = ||v2||*||u1||*cos(theta) where theta is an angle between v2 and u1 since u1 is a unit vector, ||u1|| = 1 and so v2 dot u1 = ||v2||*cos(theta) now, cos(theta) = ||proj|| / ||v2|| so v2 dot u1 = ||v2||*||proj|| / ||v2|| and thus finally v2 dot u1 = ||proj|| this is the magnitude of the projection vector we are projecting onto u1, so we can use it as a direction proj = ||proj|| * u1 / ||u1|| now again, u1 is a unit vector, so proj = ||proj|| * u1, and now TADAM! proj = (v2 dot u1) * u1

Would you end up with the same orthonormal basis if you found an orthogonal basis first, and then normalized all of the vectors at the end? It would eliminate all of that fraction multiplying.

You could normalize at the end, but during each step when you have to figure out the projection onto a subspace, having them normalized makes that calculation much easier. So pick your poison I guess.

I'm not quite sure if this is the right place to ask this question. Kindly re-direct me if there's another location where I can ask. Here it is: I have a set of 3 orthogonal vectors, and I'm in need of a fourth vector that is orthogonal to the other 3 vectors so that the 4 vectors together constitute a basis for the subspace. How do I do this? Assume for the sake of imagination that the vectors are of dimension 4.

There are a few ways to do this. I can think of two off the top of my head. The easiest would be to find the nullspace of the matrix formed by using your three vectors as columns. This will work because the nullspace is always orthogonal to the column space (the span of the column vectors.) So in this case the nullspace will be 1-dimensional and any vector in it will be orthogonal to your first three. The second way relates to Gram-Shmidt. If you can find any vector that is not in the span of the other three you can Gram-Schmidt it to make it orthogonal. This process is just like you would do for any other vector and you just project it on to the three you already have. Finding this vector might be a little tricky. Technically if you pick any vector at random you have a very good chance of picking one correctly. This is because your three vectors form a hyperplane in 4 space which is relatively small just like a plane in 3 space is relatively small.Iif you gram-schimdt it and it comes out to the zero vector you know you picked a bad vector.

At 11:30 or so, why do we ignore the normalization scalar that's being multiplied with the vectors? We can keep it aside until the end like that? Looks cool.

The Gram-Schmidt method is a way to find an orthonormal basis. To do this it is useful to think of doing two things. Given a partially complete basis we first find any vector that is orthogonal to these. Second we normalize. Then we repeat these two steps until we have filled out our basis. There are formulas that you could write down where both steps are taken care of in a long computation but it is useful to understand the process as a multiple step procedure which is what Sal is doing.

If we form the matrix A whose column vectors are the original basis "B" = {b1, b2, ..., bk}, can't we row reduce A^t to get the standard (and orthonormal) basis for its row space, which is also the standard basis for the span of our original set B? If this method works, is it better or worse for applications or otherwise?

If k=n, then indeed we will row reduce A^t to the standard (and orthonormal) basis for it's row space. Actually for k=n we don't need to work very hard to find orthonormal basis, since we know that V is the whole Rn and standard basis for Rn is also a basis for V. But if k<n, then row reduce of A^t will not result in rows of rref to form orthonormal basis for its row space. E.g. for basis vectors from this video, we will get following vectors: [1 0 0 1]^t, [0 1 0 -1]^t, [0 0 1 1]^t. These vectors will indeed span the same subspace as the original vectors, but they are not even orthogonal to each other.

Main content

Linear algebra

Unit 3: Lesson 4

Orthonormal bases and the Gram-Schmidt process

Gram-Schmidt example with 3 basis vectors

Name: Gram-Schmidt example with 3 basis vectors
Uploaded: 2011-02-20T16:47:23Z
Description: Gram-Schmidt example with 3 basis vectors

Google Classroom

Gram-Schmidt example with 3 basis vectors. Created by Sal Khan.

Sort by:

Want to join the conversation?

juha.anttila
Posted 11 years ago. Direct link to juha.anttila's post “I am puzzled. Is this no...”
I am puzzled. Is this not an example of computing in a unnecessarily complicated way? If we pick the order of vectors to be ortonormalized in a smarter(?) way :first and third are orthogonal so they just need to be normalized with (2)^(-1/2).....In the last ortogonalisation Sal realises that the first and last vectors are perpendicular, but does not comment that the whole process could/should have been done in a other order
(9 votes)
Answer
- Ryan Mackenzie Clough
  Posted 11 years ago. Direct link to Ryan Mackenzie Clough's post “You are correct, that thi...”
  You are correct, that this could be made simpler by recognizing the first and last vectors are orthogonal, and thus you can just normalize them to have two orthonormal basis vectors out of 3 right off the bat. However, the purpose of the video is to show the Graham Schmidt process from beginning to end with 3 basis vectors which can be applied to ANY set of basis vectors, not just use a trick available in this special case.
  
  The result for this example is some unnecessary computation, but this is sacrificed to provide a through and through example that can be applied to any set of 3 basis vectors.
  
  Note also that in more complex situations, it may not be immediately obvious that two/more vectors in your basis are orthogonal, so applying the Graham Schmidt processes is a rote but guaranteed way of generating an orthonormal basis.
  (21 votes)
puzzlelady7c
Posted 12 years ago. Direct link to puzzlelady7c's post “Hello Khanacademy, I unde...”
Hello Khanacademy, I understood the concept explained. I tried using the gram-schmidt method on vectors V1,V2 and V3, which have complex random variables. After the process the vectors were not orthogonal to each other. But when I tried it with real numbers it worked. Please I need more information on this. Thank you.
(8 votes)
Answer
- Niels Den Oliesjeik
  Posted 12 years ago. Direct link to Niels Den Oliesjeik's post “Some of these things don'...”
  Some of these things don't work with complex numbers, all the stuff Sal is explaining is for R^n, not C^n.
  (8 votes)
Joshua Micah Towns
Posted 11 years ago. Direct link to Joshua Micah Towns's post “This feels like a really ...”
This feels like a really dumb question...I do not understand how the projection of V2 onto V1 equals (V2 times U1) times U1. I have seen all of the videos preceding but I just can't get it. Could someone explain this to me please?
(7 votes)
Answer
- Rne Rnex
  Posted 11 years ago. Direct link to Rne Rnex's post “v2 dot u1 = ||v2||*||u1||...”
  v2 dot u1 = ||v2||*||u1||*cos(theta) where theta is an angle between v2 and u1
  since u1 is a unit vector, ||u1|| = 1 and so
  v2 dot u1 = ||v2||*cos(theta)
  now, cos(theta) = ||proj|| / ||v2|| so
  v2 dot u1 = ||v2||*||proj|| / ||v2|| and thus finally
  v2 dot u1 = ||proj||
  this is the magnitude of the projection vector
  we are projecting onto u1, so we can use it as a direction
  proj = ||proj|| * u1 / ||u1||
  now again, u1 is a unit vector, so
  proj = ||proj|| * u1, and now TADAM!
  proj = (v2 dot u1) * u1
  (10 votes)
Matthew Boulerice
Posted 10 years ago. Direct link to Matthew Boulerice's post “Would you end up with the...”
Would you end up with the same orthonormal basis if you found an orthogonal basis first, and then normalized all of the vectors at the end? It would eliminate all of that fraction multiplying.
(4 votes)
Answer
- Mitchell P
  Posted 7 years ago. Direct link to Mitchell P's post “You could normalize at th...”
  You could normalize at the end, but during each step when you have to figure out the projection onto a subspace, having them normalized makes that calculation much easier. So pick your poison I guess.
  (3 votes)
Arumugam K
Posted 10 years ago. Direct link to Arumugam K's post “I'm not quite sure if thi...”
I'm not quite sure if this is the right place to ask this question. Kindly re-direct me if there's another location where I can ask. Here it is:
I have a set of 3 orthogonal vectors, and I'm in need of a fourth vector that is orthogonal to the other 3 vectors so that the 4 vectors together constitute a basis for the subspace. How do I do this? Assume for the sake of imagination that the vectors are of dimension 4.
(3 votes)
Answer
- Lucas Van Meter
  Posted 10 years ago. Direct link to Lucas Van Meter's post “There are a few ways to d...”
  There are a few ways to do this. I can think of two off the top of my head. The easiest would be to find the nullspace of the matrix formed by using your three vectors as columns. This will work because the nullspace is always orthogonal to the column space (the span of the column vectors.) So in this case the nullspace will be 1-dimensional and any vector in it will be orthogonal to your first three.
  
  The second way relates to Gram-Shmidt. If you can find any vector that is not in the span of the other three you can Gram-Schmidt it to make it orthogonal. This process is just like you would do for any other vector and you just project it on to the three you already have. Finding this vector might be a little tricky. Technically if you pick any vector at random you have a very good chance of picking one correctly. This is because your three vectors form a hyperplane in 4 space which is relatively small just like a plane in 3 space is relatively small.Iif you gram-schimdt it and it comes out to the zero vector you know you picked a bad vector.
  (5 votes)
Aakriti Chowdhary
Posted 6 years ago. Direct link to Aakriti Chowdhary's post “Dear Sal, Will the Gram ...”
Dear Sal,
Will the Gram Schmidt process work if we apply row operations to our matrix that consists of the vectors in the basis?
If we bring our matrix to the reduced form and then calculate the characteristic polynomial?
(4 votes)
Answer
- InnocentRealist
  Posted 6 years ago. Direct link to InnocentRealist's post “That sounds pretty good! ...”
  That sounds pretty good! You can get a standard basis for the row space that way (but this isn't Gram-Schmidt).
  
  Better still: Form the matrix A whose column vectors are the original basis "B" = {b1, b2, ..., bk}, then row reduce A^t to get the standard (and orthonormal) basis for its row space, which is also the standard basis for the span of our original set B
  
  (Also, if you have k R^n basis vectors "b_i", and k<n, you can solve (A^t)x=0 (A = [b1 b2 ... bk]), then add the basis vectors for N(A) to the original set of basis vectors, then you have a basis for R^n - but this basis isn't yet orthonormal.)
  (1 vote)
ledaneps
Posted 9 years ago. Direct link to ledaneps's post “At about 1:00, can there ...”
At about
1:00
, can there be an orthogonal version of a single vector? Doesn't a vector have to be orthogonal to something? ... or have I misunderstood the concept?
(3 votes)
Answer
- Aaron Fisher
  Posted 9 years ago. Direct link to Aaron Fisher's post “It's one of those true by...”
  It's one of those true by default things. For a set of vectors to be orthogonal the dot product of any vector in the set and any other vector in the set must be zero. Since there is no other vector in the set it is vacuously true that when you do the only vector with the 'other' vectors, the dot products are never nonzero.
  (1 vote)
Ricky Mutschlechner
Posted 8 years ago. Direct link to Ricky Mutschlechner's post “At 11:30 or so, why do we...”
At
11:30
or so, why do we ignore the normalization scalar that's being multiplied with the vectors? We can keep it aside until the end like that? Looks cool.
(2 votes)
Answer
- Lucas Van Meter
  Posted 8 years ago. Direct link to Lucas Van Meter's post “The Gram-Schmidt method i...”
  The Gram-Schmidt method is a way to find an orthonormal basis. To do this it is useful to think of doing two things. Given a partially complete basis we first find any vector that is orthogonal to these. Second we normalize. Then we repeat these two steps until we have filled out our basis. There are formulas that you could write down where both steps are taken care of in a long computation but it is useful to understand the process as a multiple step procedure which is what Sal is doing.
  (2 votes)
Jonatan
Posted 9 years ago. Direct link to Jonatan's post “At 03:51, the last vector...”
At
03:51
, the last vector sal just wrote (0,0,1,1) is the vector v1, shouldn't it be the vector v2 instead (0,1,1,0)? Or is it the formula above that should be Y2=V2-Proj(V1)*u1 instead of v2 in the end?
(2 votes)
Answer
- InnocentRealist
  Posted 6 years ago. Direct link to InnocentRealist's post “The projection of v2 on v...”
  The projection of v2 on v1 is in the direction of v1, so it's magnitude is multiplied by u1 = v1/||v1||.
  (1 vote)
InnocentRealist
Posted 6 years ago. Direct link to InnocentRealist's post “If we form the matrix A w...”
If we form the matrix A whose column vectors are the original basis "B" = {b1, b2, ..., bk}, can't we row reduce A^t to get the standard (and orthonormal) basis for its row space, which is also the standard basis for the span of our original set B?

If this method works, is it better or worse for applications or otherwise?
(1 vote)
Answer
- bochenec
  Posted 5 years ago. Direct link to bochenec's post “If k=n, then indeed we wi...”
  If k=n, then indeed we will row reduce A^t to the standard (and orthonormal) basis for it's row space. Actually for k=n we don't need to work very hard to find orthonormal basis, since we know that V is the whole Rn and standard basis for Rn is also a basis for V.
  
  But if k<n, then row reduce of A^t will not result in rows of rref to form orthonormal basis for its row space. E.g. for basis vectors from this video, we will get following vectors:
  [1 0 0 1]^t, [0 1 0 -1]^t, [0 0 1 1]^t. These vectors will indeed span the same subspace as the original vectors, but they are not even orthogonal to each other.
  (2 votes)

Video transcript

Let's do one more Gram-Schmidt example. So let's say I have the subspace V that is spanned by the vectors-- let's say we're dealing in R4, so the first vector is 0, 0, 1, 1. The second vector is 0, 1, 1, 0. And then a third vector-- so it's a three-dimensional subspace of R4-- it's 1, 1, 0, 0, just like that, three-dimensional subspace of R4. And what we want to do, we want to find an orthonormal basis for V. So we want to substitute these guys with three other vectors that are orthogonal with respect to each other and have length 1. So we do the same drill we've done before. We can say-- let's call this guy v1, this guy is v2, and let's call this guy v3. So the first thing we want to do is replace v1-- and I'm just picking this guy at random because he was the first guy on the left-hand side. I want to replace v1 with an orthogonal version of v1. So let me call u1 is equal to-- well, let me just find out the length the v1. I don't think I have to explain too much of the theory at this point. I just want to show another example. So the length of v1 is equal to the square root of 0 squared plus 0 squared plus 1 squared plus 1 squared, which equals the square root of 2. So let me define my new vector u1 to be equal to 1 over the length of v1, 1 over the square root of 2, times v1, times 0, 0, 1, 1. And just like that, the span of v1, v2, v3, is the same thing is the span of u1, v2, and v3. So this is my first thing that I've normalized. So I can say that V is now equal to the span of the vectors u1, v2, and v3. Because I can replace v1 with this guy, because this guy is just a scaled-up version of this guy. So I can definitely represent him with him, so I can represent any linear combination of these guys with any linear combination of those guys right there. Now, we just did our first vector. We just normalized this one. But we need to replace these other vectors with vectors that are orthogonal to this guy right here. So let's do v2 first. So let's replace-- let's call it y2 is equal to v2 minus the projection of v2 onto the space spanned by u1 or onto-- you know, I could call it c times u1, or in the past videos, we called that subspace V1, but the space spanned by u1. And that's just going to be equal to y2 is equal to v2, which is 0, 1, 1, 0, minus-- v2 projected onto that space is just a dot product of v2, 0, 1, 1, 0, with the spanning vector of that space. And there's only one of them, so we're only going to have one term like this with u1, so dotted with 1 over the square root of 2 times 0, 0, 1, 1, and then all of that times u1. So 1 over the square root of 2 times the vector 0, 0, 1, 1. And so this is going to be equal to v2, which is 0, 1, 1, 0. The square root of 2, let's factor them out. So then you just get-- or kind of reassociate them out. So then you get this is 1 over the square root of 2 times 1 over the square root of 2 is minus 1/2. You times-- what's the dot product of these two guys? You get 0 times 0 plus 1 times 0, which is still 0, plus 1 times 1 plus 0 times 0. So you're just going to have times 1 times this out here: 0, 0, 1, 1. I'll write that a little bit neater. I'm getting careless. 1, 1. So this is just going to be equal to 0, 1, 1, 0 minus-- 1/2 times 0 is 0. 1/2 times 0 is 0. Then I have two halves here. So y2 is equal to-- let's see, 0 minus 0 is 0, 1 minus 0 is 1, 1 minus 1/2 is 1/2, and then 0 minus 1/2 is minus 1/2. So V, we can now write as the span of u1, y2, and v3. And this is progress. u1 is orthogonal, y2-- sorry, u1 is normalized. It has length 1. Y2 is orthogonal to it or they're orthogonal with respect to each other, but y2 still has not been normalized. So let me replace y2 with a normalized version of it. The length of y2 is equal to the square root of 0 plus 1 squared, which is 1, plus 1/2 squared, which is 1/4, plus minus 1/2 squared, which is also 1/4, so plus 1/4. So this is 1 and 1/2. So it's equal to the square root of 3/2. So let me define another vector here. u2, which is equal to 1 over the square root of 3/2, or we could say is the square root of 2/3, I'm just inverting it. It's 1 over the length of y2. So I'll just find the reciprocal, so it's the square root 2 over 3 times y2, times this guy right here, times 0, 1, 1/2, and minus 1/2. And so this span is going to be the same thing as the span of u1, u2, and v3. And there's our second basis vector. And we're making a lot of progress. These guys are orthogonal with respect to each other. They both have length 1. We just have to do something about v3. And we do it the same way. Let's find a vector that is orthogonal to these guys, and if I sum that vector to some linear combination of these guys, I'm going to get v3, and I'm going to call that vector y3. y3 is equal to v3 minus the projection of v3 onto the subspace spanned by u1 and u2. So I could call that subspace-- let me just write it here. The span of u1 and u2, just for notation, I'm going to call it v2. So it's v3, and actually, I don't even have to write that . Minus the projection of v3 onto that. And what's that going to be? That's going to be v3 dot u1 times u1, times the vector u1. And actually let me just-- plus v3 dot u2 times the vector u2. Since this is an orthonormal basis, the projection onto it, you just take the dot product of v2 with each of their orthonormal basis vectors and multiply them times the orthonormal basis vectors. We saw that several videos ago. That's one of the neat things about orthonormal bases. So what is this going to be equal to? A little bit more computation here. y3 is equal to v3, which was up here. That's v3. v3 looks like this. It's 1, 1, 0, 0 minus v3 dot u1. So this is minus v3, 1, 1, 0, 0, dot u1. So it's dot 1 over the square root of 2 times 0, 0, 1, 1. That's u1-- so that's this part right here-- times u1, so times 1 over the square root of 2 times 0, 0, 1, 1. This piece right there is this piece right there. And then we can distribute this minus sign, so it's going to be plus. You know, we have a plus, but there's this minus over here so we put a minus v3. Let me switch colors . Minus v3 , which is 1, 1 0, 0 dotted with u2, dotted with the square root of 2/3 times 0, 1, 1/2, minus 1/2 times u2, times the vector u2, times the square root of 2/3, times the vector 0, 1, 1/2, minus 1/2. And what do we get? Let's calculate this. So we could take the-- so this is going to be equal to the vector 1, 1, 0, 0, minus-- so the 1 over the square root of 2 and the 1 over the square root of 2, multiply them. You're going to get a 1/2. And then when you take the dot product of these two, 1 times 0-- let's see, this is actually all going to be, if you take the dot product of all of these, then it actually gets 0, right? So this guy, v3, was actually already orthogonal to u1. This will just go straight to 0, which is nice. We don't have to have a term right there. I took the dot product 1 times 0 plus 1 times 0 plus 0 times 1 plus 0 times 1, all gets zeroed. So this whole term drops out. We can ignore it, which makes our computation simpler. And then over here we have minus the square root of 2/3 times the square root of 2/3 is just 2/3 times the dot product of these two guys. So that's 1 times 0, which is 0, plus 1 times 1, which is 1, plus 0 times 1/2, which is 0, plus 0 times minus 1/2, which is 0, so we just get a 1 there, times the vector 0, 1, 1/2, minus 1/2. And then what do we get? We get-- this is the home stretch-- 1, 1, 0, 0 minus 2/3 times all of these guys. So 2/3 time 0 is 0. 2/3 times 1 is 2/3. 2/3 times 1/2 is 1/3. And then 2/3 times minus 1/2 is minus 1/3. So then this is going to be equal to 1 minus 0 is 1, 1 minus 2/3 is 1/3, 0 minus 1/3 is minus 1/3, and then 0 minus minus 1/3 is positive 1/3. So this vector y3 is orthogonal to these two other vectors, which is nice, but it still hasn't been normalized. So we finally have to normalize this guy, and then we're done. Then we have an orthonormal basis. We'll have u1, u2, and now we'll find u3. So the length of my vector y-- actually, let's do something even better. It'll simplify things a little bit. Instead of a writing y this way, I could scale up y, right? All I want is a vector that's orthogonal to the other two that still spans the same space. So I can scale this guy up. So I could say, I don't know, let me call it y3-- let me call it y3 prime. And I'm just doing this to ease the computation. I could just scale this guy up, multiply him by 3. So what do I get? I probably should have done it some of the other ones. 3, 1, minus 1, and 1. And so I can replace y3 with this guy, and then I can just normalize this guy. It'll be a little bit easier. So the length of y3 prime that I just defined is equal to the square root of 3 squared, which is 9, plus 1 squared plus minus 1 squared plus 1 squared, which is equal to the square root of 12, which is what? That's two square roots of 3. That is equal to 2 square roots of 3, right? Square root of 4 times the square root of 3, which is two square roots of 3. So now I can to find u3 as equal to y3 times 1 over the length of y3, so it's equal to 1 over two square roots of 3 times the vector 3, 1, minus 1, and 1. And then we're done. If we have a basis, an orthonormal basis would be this guy-- let me take the other ones down here-- and these guys. All of these form-- let me bring it all the way down. If I have a collection of these three vectors, I now have an orthonormal basis for V, these three right there. That set is an orthonormal basis for my original subspace V that I started off with.

Creative Commons Attribution/Non-Commercial/Share-Alike Video on YouTube