Main content
Course: Statistics and probability > Unit 5
Lesson 5: Assessing the fit in least-squares regression- Residual plots
- Residual plots
- R-squared intuition
- R-squared or coefficient of determination
- Standard deviation of residuals or Root-mean-square error (RMSD)
- Interpreting computer regression data
- Interpreting computer output for regression
- Impact of removing outliers on regression lines
- Effects of influential points
© 2024 Khan AcademyTerms of usePrivacy PolicyCookie Notice
R-squared intuition
When we first learned about the correlation coefficient, , we focused on what it meant rather than how to calculate it, since the computations are lengthy and computers usually take care of them for us.
We'll do the same with and concentrate on how to interpret what it means.
In a way, measures how much prediction error is eliminated when we use least-squares regression.
Predicting without regression
We use linear regression to predict given some value of . But suppose that we had to predict a value without a corresponding value.
Without using regression on the variable, our most reasonable estimate would be to simply predict the average of the values.
Here's an example, where the prediction line is simply the mean of the data:
Notice that this line doesn't seem to fit the data very well. One way to measure the fit of the line is to calculate the sum of the squared residuals—this gives us an overall sense of how much prediction error a given model has.
So without least-squares regression, our sum of squares is
Would using least-squares regression reduce the amount of prediction error? If so, by how much? Let's see!
Predicting with regression
Here's the same data with the corresponding least-squares regression line and summary statistics:
Equation | ||
---|---|---|
This line seems to fit the data pretty well, but to measure how much better it fits, we can look again at the sum of the squared residuals:
Using least-squares regression reduced the sum of the squared residuals from to .
So using least-squares regression eliminated a considerable amount of prediction error. How much though?
R-squared measures how much prediction error we eliminated
Without using regression, our model had an overall sum of squares of . Using least-squares regression reduced that down to .
So the total reduction there is .
We can represent this reduction as a percentage of the original amount of prediction error:
If you look back up above, you'll see that .
R-squared tells us what percent of the prediction error in the variable is eliminated when we use least-squares regression on the variable.
As a result, is also called the coefficient of determination.
Many formal definitions say that tells us what percent of the variability in the variable is accounted for by the regression on the variable.
It seems pretty remarkable that simply squaring gives us this measurement. Proving this relationship between and is pretty complex, and is beyond the scope of an introductory statistics course.
Want to join the conversation?
- Which parameter is then better to evaluate the fit of a line to a data set? the correlation coefficient (r) or the coefficient of determination (r2)?(26 votes)
- The short answer is this: In the case of the Least Squares Regression Line, according to traditional statistics literature, the metric you're looking for is r^2.
Longer answer:
IMHO, neither r o r^2 are the best for this. In the case of r, it is calculated using the Standard Deviation, which itself is a statistic that has been long put to doubt because it squares numbers just to remove the sign and then takes a square root AFTER having added those numbers, which resembles more an Euclidean distance than a good dispersion statistic (it introduces an error to the result that is never fully removed). Here is a paper about that topic presented at the British Educational Research Association Annual Conference in 2004: https://www.leeds.ac.uk/educol/documents/00003759.htm .
If we used the MAD (mean absolute deviation) instead of the standard deviation to calculate both r and the regression line, then the line, as well as r as a metric of its effectiveness, would be more realistic, and we would not even need to square r at all.
This is a very extensive subject and there are still lots of different opinions out there, so I encourage other people to complement my answer with what they think.
Hope you found my answer helpful or at least interesting.
Cheers!(115 votes)
- what's the difference between R-squared and the total sum of squared residual?(9 votes)
- They lost me at the squares(6 votes)
- don't worry about them too much
they're simply a visualization of squaring numbers then summing them like 3^2 + 7^2 + 13^2 to assess how far they are from a regression line(5 votes)
- If you have two models of a set of data, a linear model and a quadratic model, and you have worked out the R-squared value through linear regression, and are then asked to explain what the R-squared value of the quadratic model is, without using any figures, what would this explanation be?(2 votes)
- A quadratic model has one extra parameter (the coefficient on x^2) compared to a linear model. Therefore, the quadratic model is either as accurate as, or more accurate than, the linear model for the same data. Recall that the stronger the correlation (i.e. the greater the accuracy of the model), the higher the R^2. So the R^2 for the quadratic model is greater than or equal to the R^2 for the linear model.
Have a blessed, wonderful day!(5 votes)
- How we predict sum of squares in the regression line?(2 votes)
- Tbh, you really cannot get around squaring every number. I guess if you have decimals, you could round them them off, but really,, other than that, there’s no shortcut. It is difficult to predict because the powers have to be applied to each and every number. You could always do a bit of mental math and round things off into easier numbers, but it’s not always reliable.(5 votes)
- Why do we square the residuals? I get that we need a positive value for all residuals to calculate the sum of the prediction error, but wouldn't it be easier to just calculate the sum of the absolute values of the residuals?(3 votes)
- is there a shorter way to create a estimation without taking all the steps to solve the problem? is there a hack or a way to do it quickly?(2 votes)
- Maybe using software like JASP?(1 vote)
- Can I get the exact data set, based on that this dot plot have been created.(2 votes)
- Hi. I have in several cases lately observed that when you remove several obvious outliers from a data set in "one go" the R-sq actually gets lower? This is rather counter intuitive to me at least, and also when looking into the formula for how R is calculated it doesn't seem to make much sense either? Any insights into this?
b/r
Niels(2 votes)- The phenomenon you described, where removing outliers from a dataset results in a lower R^2 value, can occur in certain cases. One possible reason is that the outliers were exerting a disproportionate influence on the correlation between the variables, causing the regression line to be biased. Removing outliers may result in a more accurate estimation of the true relationship between the variables, leading to a lower residual sum of squares and hence a lower R^2 value. Additionally, it's important to consider the nature of the outliers and the underlying data generating process to fully understand the impact of their removal on the regression analysis.(1 vote)
- how do you calculate r^2(1 vote)
- in case you already have r, simply do r*r
yes. r^2 is nothing but a square of r(correlation coefficient)(3 votes)