I have one question. Is A (A^T A) ^(-1) A^T equal to A A^(-1) A^T^(-1) A^T and equal to I I and equal to I? I am confused with the projection matrix. Thank you in advance!! ^^

The property (AB)^-1=(B)^-1*(A)^-1 is valid only when both A and B are invertible and when matrix multiplication between them is defined. If A is invertible, then it follows that A^T is also invertible. Their product A^T A is defined because the number of rows in A^T is equal to the number of columns in A. In such a case, the simplification A (A^T A) ^(-1) A^T =A A^(-1) A^T^(-1) A^T=I would be valid. So the projection of x onto the column space is simply x. In fact, this makes since because when A is invertible, the system Ax=b has a unique solution for every b in Rn. This implies that the columns of A are a basis for Rn(since they are linearly independent and they span Rn) and that therefore any projection of an arbitrary vector x onto the subspace spanned by the columns of A is simply x, since x is already in the columns space of A. However, If A is not invertible, then apparently there are some elements of Rn that are not in the column space of A, and so it makes since to speak of the projection of arbitrary vectors into C(A), which can be computed using the projection matrix A (A^T A) ^(-1) A^T. Keep in mind that even if A itself is not invertible, A^T A is invertible since A consists of linearly independent columns.

So the final transformation matrix at 12:07 is symmetric. Is there anything significant about this? Will it always be symmetric?

Yes it will be symmetric. If A is axb, then A' is bxa. So A'A is bxb inverse(A'A) is still bxb... A * inverse(A'A) * A' gives axa matrix. Since in this case we are dealing with R4, we expect a vector of R4 as input so the final transformation matrix has 4 columns. In general the projection will be a vector in R4 so the matrix is 4x4. But the interesting thing here is that the 3rd row is zero. So the projection matrix takes a vector in R4 and returns a vector in R4 whose 3rd component is 0 (so it is kind of like in R3). Why is the 3rd row all zeroes? note that all the basis of V have zeroes in the 3rd position, So this subspace can span vectors that have components in 1s, 2nd and 4th dimension only. So if you project something onto V, you are bound to lose the 3rd component (9which is what we find from the matrix which has 3rd row full of zeroes only). To conclude: yes the transformation matrix is symmetric as it is a general matrix that transforms R4 vectors to R4... but some components of the output vector might be zero making them vectors of a smaller subspace.

Main content

Linear algebra

Course: Linear algebra > Unit 3

Lesson 2: Orthogonal projections

Subspace projection matrix example

Name: Subspace projection matrix example
Uploaded: 2011-02-20T16:47:21Z
Description: Example of a transformation matrix for a projection onto a subspace

Google Classroom

Example of a transformation matrix for a projection onto a subspace. Created by Sal Khan.

Want to join the conversation?

Sort by:

srmoon07
Posted 12 years ago. Direct link to srmoon07's post “I have one question. ...”
I have one question.

Is A (A^T A) ^(-1) A^T equal to A A^(-1) A^T^(-1) A^T and equal to I I and equal to I?

I am confused with the projection matrix.

Thank you in advance!! ^^
Button navigates to signup pageButton navigates to signup page
(9 votes)
Answer
- Aaron Williams
  Posted 8 years ago. Direct link to Aaron Williams's post “The property (AB)^-1=(B)^...”
  The property (AB)^-1=(B)^-1*(A)^-1 is valid only when both A and B are invertible and when matrix multiplication between them is defined.
  
  If A is invertible, then it follows that A^T is also invertible. Their product A^T A is defined because the number of rows in A^T is equal to the number of columns in A. In such a case, the simplification A (A^T A) ^(-1) A^T =A A^(-1) A^T^(-1) A^T=I would be valid. So the projection of x onto the column space is simply x.
  
  In fact, this makes since because when A is invertible, the system Ax=b has a unique solution for every b in Rn. This implies that the columns of A are a basis for Rn(since they are linearly independent and they span Rn) and that therefore any projection of an arbitrary vector x onto the subspace spanned by the columns of A is simply x, since x is already in the columns space of A.
  
  However, If A is not invertible, then apparently there are some elements of Rn that are not in the column space of A, and so it makes since to speak of the projection of arbitrary vectors into C(A), which can be computed using the projection matrix A (A^T A) ^(-1) A^T. Keep in mind that even if A itself is not invertible, A^T A is invertible since A consists of linearly independent columns.
  Button navigates to signup page
  (7 votes)
wchargin
Posted 9 years ago. Direct link to wchargin's post “So the final transformati...”
So the final transformation matrix at
12:07
is symmetric. Is there anything significant about this? Will it always be symmetric?
Button navigates to signup pageButton navigates to signup page
(2 votes)
Answer
- Sayantan Sarkar
  Posted 8 years ago. Direct link to Sayantan Sarkar's post “Yes it will be symmetric....”
  Yes it will be symmetric.
  If A is axb, then A' is bxa. So A'A is bxb inverse(A'A) is still bxb... A * inverse(A'A) * A' gives axa matrix.
  Since in this case we are dealing with R4, we expect a vector of R4 as input so the final transformation matrix has 4 columns. In general the projection will be a vector in R4 so the matrix is 4x4. But the interesting thing here is that the 3rd row is zero. So the projection matrix takes a vector in R4 and returns a vector in R4 whose 3rd component is 0 (so it is kind of like in R3).
  Why is the 3rd row all zeroes? note that all the basis of V have zeroes in the 3rd position, So this subspace can span vectors that have components in 1s, 2nd and 4th dimension only. So if you project something onto V, you are bound to lose the 3rd component (9which is what we find from the matrix which has 3rd row full of zeroes only).
  
  To conclude: yes the transformation matrix is symmetric as it is a general matrix that transforms R4 vectors to R4... but some components of the output vector might be zero making them vectors of a smaller subspace.
  Comment on Sayantan Sarkar's post “Yes it will be symmetric....”
  (2 votes)
Mehmoodd.alam
Posted 7 years ago. Direct link to Mehmoodd.alam's post “How can we find projectio...”
How can we find projection of a vector in R4 on R2 ?
Button navigates to signup pageButton navigates to signup page
(1 vote)
Answer
- Kyler Kathan
  Posted 7 years ago. Direct link to Kyler Kathan's post “R2 isn't a subspace of R4...”
  R2 isn't a subspace of R4, it's an entirely separate vector space; so you can't. However, if you're asking how we can find the projection of a vector in R4 onto the plane spanned by the î and ĵ basis vectors, then all you need to do is take the [x y z w] form of the vector and change it to [x y 0 0]. For example:
  S = span(î, ĵ)
  v = [2 3 7 1]
  proj(v onto S) = [2 3 0 0]
  Comment on Kyler Kathan's post “R2 isn't a subspace of R4...”
  (1 vote)
Ryan Jin
Posted 4 years ago. Direct link to Ryan Jin's post “What if you try to projec...”
What if you try to project a vector in R5 onto this subspace? How would you multiply the 1x5 vector by a 4x4 matrix
Button navigates to signup pageComment on Ryan Jin's post “What if you try to projec...”
(1 vote)
Answer
cOOPER hAD cAPS lOCKED oN
Posted 4 months ago. Direct link to cOOPER hAD cAPS lOCKED oN's post “Does anyone have an good ...”
Does anyone have an good cookie recipes?
Button navigates to signup pageButton navigates to signup page
(0 votes)
Answer

Video transcript

Let's say I've got some subspace V, which tends to be our favorite letter for subspaces, and it's equal to the span of two vectors in R4. Let's say that the first vector is 1 0 0 1, and the second vector is 0 1 0 1. That is my subspace V. And you can see that these are going to be a basis. That these are linearly independent. Two vectors that are linear-- or any set of vectors that are linearly independent and that span a subspace are a basis for that subspace. You can see they are linearly independent. This guy's has a 1 here. There's no way you can take some combination of this guy to somehow get a 1 there. And this guy has a 1 here. There's no way you can get some linear combination of these zeroes here to a 1 there, so they're linearly dependent. You can also call this a basis for V. Now, given that, let's see if we can find out the transformation matrix for the projection of any arbitrary vector onto this subspace. So let's say that X-- we're dealing in R4 here, right? Let's say that x is a member of R4, and I want to figure out a transformation matrix for the projection onto V of x. Now, in the last video, we came up with a general way to figure this out. We said if A is a transformation matrix-- sorry. If A is a matrix who's columns are the basis for the subspace, so let's say A is equal to 1 0 0 1, 0 1 0 1. So A is a matrix whose columns are the basis for our subspace, then the projection of x onto V would be equal to-- and this is kind of hard. The first time you look at, it gives you a headache, but there's a certain pattern or symmetry or a way of-- you could say it's A times, you're gonna have something in the middle, and then you have A transpose times your vector x. And the way I remember it is in the middle, you have these two guys switched around. So then you have A transpose A, and you take the inverse of it. You probably won't be using this in your everyday life five or ten years from now, so it's OK if you don't memorize it, but temporarily, put this in your medium-term memory because it's a good thing to know for doing these projection problems. So if we want to find the general matrix for this transformation, we just have to determine what this matrix is equal to, and that's just a bunch of matrix operations. So that's A. What is A transpose? A transpose is going to be equal to just all the rows turn into columns. So the first column becomes the first row. So it becomes 1 0 0 1. The second column becomes the second row 0 1 0 1. That's what A transpose is. Now, what's A transpose A? To figure out that, I want to figure out what A transpose times A is. So let me multiply A transpose times A. So I'll rewrite A right here. 1 0 0 1, 0 1 0 1. This is giving us some good practice on matrix-matrix products. This is going to be equal to what? Well, first of all, this is a 2-by-4 matrix, and I'm multiplying it by a 4-by-2 matrix, so it's going to be a 2-by-2 matrix. So the first entry is essentially the dot product of that row with that column. So it's 1 times 1 plus 0 times 0 plus 0 times 0 plus 1 times 1. So it's just going to be 2 for that first entry right there. And then you take the dot product of this guy with this guy right here. So it's 1 times 0, which is 0, plus 0 times 1, which is 0, plus 0 time 0, which is 0, plus 1 times 1, which is 1. Now, we do this guy dotted with this column right there. 0 times 1 is 0 plus 1 time 0 is 0 plus 0 times 0 is 0 plus 1 times 1 is 1. And then finally, this row dotted with this second column. Second row, second column. 0 times 0 is 0, 1 times 1 is 1, 0 times 0 is 0, 1 times 1 is 1. So we have 1 times 1 plus 1 times 1. It's going to be 2. It's going to be equal to 2. So this right here is A transpose A. But that's not good enough. We need to figure out what the inverse of A transpose A is. This is A transpose A. But we need to figure out A transpose A inverse. So what's the inverse of this? So let me write it here. The inverse A transpose A inverse is going to be equal to what? It's 1 over the determinant of this guy. What's the determinant here? It's going to be 1 over the determinant of this. The determinant is 2 times 2, which is 4, minus 1 times 1. So it's 4 minus 1, which is 3. So 1 over the determinant times this guy, where if I swap these two, so I swap the 1's-- sorry, I swap the 2's. So this 2 goes here, and then this orange 2 goes over here. And then I make these 1's negative. This becomes a minus 1 and this becomes a minus 1. We learned that this is a general solution for the inverse of a 2-by-2 matrix. I think it was 10 or 11 videos ago, and you probably learned this in your Algebra II class, frankly, but there you go. We have A transpose A inverse. So we have this guy. We have this whole guy here is just this matrix. I could multiply the 1/3 into it, but I don't have to do that just yet. But let's figure out the whole matrix now. The whole A times this guy, A transpose A inverse times A transpose. Let me write it this way. So the projection onto the subspace V of x is going to be equal to A. 1 0 0 1-- let me write a little bit bigger like this. So 1 0 0 1, 0 1 0 1 times A transpose A inverse, right? A times A transpose A inverse, which is this guy right here. Let's just put the 1/3 out front just because that's just a scalar. I'll put the 1/3 out front times this guy. This A transpose A inverse is 1/3 times 2 minus 1, minus 1, 2. And then I'm going to multiply it times A transpose. And all that times our vector x. So A transpose is right there. It is 1 0 0 1, 0 1 0 1. And then all of that's going to be times your vector x. So we still have some nice matrix-matrix products ahead of us. Let see if we can do these. So the first one, let's just multiply these two guys. I don't think there's any simple way to do it. This is a 2-by-2 matrix and this is a 2-by-4 matrix, so when I multiply them, I'm going to end up with a 2-by-4 matrix. Let me write that 2-by-4 matrix right here. And then I can write this guy right here. 1 0 0 1, 0 1 0 1. And then I have the 1/3 that was from A transpose A inverse, but I put the scaling factor out there. And all of this is equal to the projection of x onto V. So let's do this product. So this first entry is going to be 2 times 1 plus minus 1 times 0, so that is just 2. Then you're going to have 2 times 0 plus minus 1 times 1. Well, that's minus 1. Then you have 2 times 0 plus minus 1 times 0. Well, that's just 0. And then you're going to have 2 times 1 plus minus 1 times 1. That's 2 minus 1. That's just 1, right? 2 times 1 plus minus 1 times 1. Fair enough. Now, let's do the second row. Minus 1 times 1 plus 2 times 0, so that's just minus 1. Minus 1 times 0 plus 2 times 1. Well, that's just 2. Minus 1 times 0 plus 2 times 0. That's just 0. Minus 1 times 1 plus 2 times 1. Well, that's minus 1 plus 2, so that is 1. Almost there, and, of course, we have to multiply it times x at the end. That's what the transformation is. But this right here is our transformation matrix. One more left to do. Let's hope I haven't made any careless mistakes and that I won't make any when doing this product. This is going to be a little more complicated because this is a 4 by 2 times a 2 by 4. I'm going to end up with a 4-by-4 matrix. Let me give myself some breathing room here because I'm going to generate a 4-by-4 matrix right there. And so what am I going to get? So this first entry is going to be 1 times 2 plus 0 times minus 1. So it's just going to be equal to 2. The next entry: 1 times-- this row times any column here is just going to be the first entry in the column because it gets zeroed out. So 1 times 2 plus 0 times minus 1 is just 2. 1 times minus 1 plus 0 times 2 is just minus 1. 1 times 0 plus 0 times 0 is 0. 1 times 1 plus 0 times 1 is just 1. When you take this row and you multiply it times these columns, you literally just got your first row there. Now, let's do this row times these columns. Now, you've got a 0 here, so you're going to have a 0 times the first entry of all of these and a 1 times the second one. So 0 times 2 plus 1 times minus 1 is minus 1. 0 times minus 1 plus 1 times 2 is 2. You're just going to get the second row here. 2 0 1. That actually makes sense, because if you just look at this part of the matrix, it's the 2-by-2 identity matrix. So, anyway, that's a little hint why this looks very much like that, but we're just going to go through this matrix product. Now, you multiply this-- let me do it in a different color. You multiply this guy times each of these columns. That guy dotted with that is just going to 0 because this guy's essentially the 0 row vector, so you're just going to get a bunch of zeroes. And then, finally, this last row, it's 1 times the first entry plus 1 times the second entry. So this guy's going to be 2 plus minus 1, which is 1. Minus 1 plus 2, which is 1. 0 plus 0, which is 0. And then 1 plus 1, which is 2. And all that times x. And there you have it. This is exciting! The projection onto V of x is equal to this whole matrix times x. So this thing right here, I could multiply the 1/3 into it, but we don't have to do that. That'll just make it a little bit more messy. This thing right here is the transformation matrix. As you can see, since we're transforming-- remember, this projection onto V, this is a linear transformation from R4 to R4. You give me some member of R4, and I'll give you another member of R4 that's in my subspace that is the projection. So this is going to be a 4-by-4 You can see it right there. Anyway, hopefully, you found that useful to actually see a tangible result. R4 is very abstract, so this would even be beyond our three-dimensional programming example. We're dealing with a more abstract data set where we're interested in finding a projection.