- Introduction to Type I and Type II errors
- Examples identifying Type I and Type II errors
- Type I vs Type II error
- Introduction to power in significance tests
- Examples thinking about power in significance tests
- Error probabilities and power
- Consequences of errors and significance
Both type 1 and type 2 errors are mistakes made when testing a hypothesis. A type 1 error occurs when you wrongly reject the null hypothesis (i.e. you think you found a significant effect when there really isn't one). A type 2 error occurs when you wrongly fail to reject the null hypothesis (i.e. you miss a significant effect that is really there).
Want to join the conversation?
- Why is P(type I error) = significance level? What's the logic behind it?(7 votes)
- Great question! I hope I can do it justice.
When we choose a significance level, we're saying that we're willing to accept a Type I error occurring with that much probability or, in other words, "that often."
The reason they're the same thing is, when performing a significance/hypothesis test, we are comparing the probability of the outcome we get from our sample (as well as those less likely [altogether, our p-value]) to the significance level that we had set, and that is how we're going to make our choice to reject or fail to reject the null hypothesis.
In other words, the significance level is a probability threshold. Any outcome with a probability less than that threshold will cause us to reject the null hypothesis, regardless of whether it’s true.
In the case when it’s true, that is how often we would be committing a Type I error.
For example, if we set a significance level of 5%, that means we will reject the null hypothesis every time our p-value is less than 5%. If the null hypothesis is true, our p-value will be less than 5% roughly 5% of the times we do the test, and then we will reject the null hypothesis by mistake 5% of the time, and so our Type I error rate (another name for significance level, or alpha) is 5%.
Also, if our significance level is high, then many of the possible outcomes will cause us to reject the null hypothesis; likewise, if our significance level is low, then few of the possible outcomes will cause us to reject the null hypothesis. If the null hypothesis is actually correct, then in all of those cases, we would have committed a Type I error.
Hope this helps. :)(23 votes)
- give a significant level of 0.05, then the chance of rejecting H0 when H0 is true is 0.05, so the chance of fail to reject H0 when H0 is true is 0.95. But what is the probability of rejecting H0 when H0 is false, and what is the probability of fail to reject H0 when H0 is false.(6 votes)
- Wanted to understand how this is related to confusion matrix calculated while doing machine learning (Python specifically) , as the 'correct entries' are in the wrong diagonal in this case.(1 vote)
- You are right, in a confusion matrix, ground truth values are along the rows and predicted values along the columns. I think it's just a convention difference. Type I error is still false positive and Type II is still false negative.(1 vote)
- A Type I error occurs when we reject the null hypothesis of a population parameter when the null hypothesis is actually true. But how do we know that the null hypothesis is true, considering that we can never be certain about a population parameter?(1 vote)
- [Instructor] What we're gonna do in this video is talk about Type I errors and Type II errors and this is in the context of significance testing. So just as a little bit of review, in order to do a significance test, we first come up with a null and an alternative hypothesis. And we'll do this on some population in question. This will say some hypotheses about a true parameter for this population. And the null hypothesis tends to be kind of what was always assumed or the status quo while the alternative hypothesis, hey, there's news here, there's something alternative here. And to test it, and we're really testing the null hypothesis. We're gonna decide whether we want to reject or fail to reject the null hypothesis, we take a sample. We take a sample from this population. Using that sample, we calculate a statistic, we calculate a statistic, that's trying to estimate the parameter in question. And then using that statistic, we try to come up with the probability of getting that statistic, the probability of getting that statistic that we just calculated from that sample of a certain size, given if we were to assume that our null hypothesis, if our null hypothesis is true. And if this probability, which is often known as a p-value, is below some threshold that we set ahead of time which is known as the significance level, then we reject the null hypothesis. Let me write this down. So this right over here, this is our p-value. This should all be review, we introduced it in other videos. We have seen on other videos if our p-value is less than our significance level, then we reject our null hypothesis, and if our p-value is greater than or equal to our significance level, alpha, then we fail to reject, fail to reject our null hypothesis. And when we reject our null hypothesis, some people will say that might suggest the alternative hypothesis. And the reason why this makes sense is if the probability of getting the statistic from a sample of a certain size, if we assume that the null hypothesis is true is reasonably low if it's below a threshold, maybe this threshold is 5%, if the probability of that happening was less than 5%, then hey, maybe it's reasonable to reject it. But we might be wrong in either of these scenarios and that's where these errors come into play. Let's make a grid to make this clear. So there's the reality, let me put reality up here, so the reality is there's two possible scenarios in reality, one is the null hypothesis is true and the other is that the null hypothesis is false, and then based on our significance test, there's two things that we might do, we might reject the null hypothesis, or we might fail to reject the null hypothesis. And so let's put a little grid here to think about the different combinations, the different scenarios here. So in a scenario where the null hypothesis is true, but we reject it, that feels like an error. We shouldn't reject something that is true and that indeed is a Type I error. Type I error. You shouldn't reject the null hypothesis if it was true. And you can even figure out what is the probability of getting a Type I error. Well that's gonna be your significance level because if your null hypothesis is true, let's say that your significance level is 5%, well 5% of the time, even if your null hypothesis is true, you're going to get a statistic that's going to make you reject the null hypothesis. So one way to think about the probability of a Type I error is your significance level. Now, if your null hypothesis is true and you failed to reject it, well that's good. This we can write this as, this is a correct conclusion. The good thing just happened to happen this time. Now, if your null hypothesis is false and you reject it, that's also good. That is the correct conclusion. But if your null hypothesis is false and you failed to reject it, well then that is a Type II error. That is a Type II error. Now with this context, in the next few videos, we will actually do some examples where we try to identify, one, whether an error is occurring and whether that error is a Type I or a Type II.