If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

## Statistics and probability

### Course: Statistics and probability>Unit 12

Lesson 1: The idea of significance tests

# P-values and significance tests

We compare a P-value to a significance level to make a conclusion in a significance test. Given the null hypothesis is true, a p-value is the probability of getting a result as or more extreme than the sample result by random chance alone. If a p-value is lower than our significance level, we reject the null hypothesis. If not, we fail to reject the null hypothesis.

## Want to join the conversation?

• why do we reject the H(o) when p value is less than the significance level?
• Because we're looking for the probability that the sample mean (X bar) is greater than or equal to 25 minutes. if we assume the null hypothesis to be true, then the p-value would display the percent chance of getting the result if the null hypothesis were true. If the chance is lower than our significance level (1 in 20 or .05 in this case), then that's evidence that such an outcome would be rather unlikely to occur if the null hypothesis were true.
• In the case of sampling from the yellow version, I don't understand why we can assume the null hypothesis to be equal to 20 minutes. The null hypothesis being equal to 20 minutes doesn't seem to have any relationship to the p-value to me. Can someone help explain?
• A mean of 20 conveys that there is no difference. We expect that any new sample of users who use the yellow-background website will spend, on average, the same amount of time as they would have on the off-white-background website.

That is, if the null hypothesis were true, then every sample of 100 that we take from the population of users who visit the yellow-background website should have a mean close to 20.

The p-value tells us how likely it is to get a sample mean of 25 when the sample mean should be close to 20.

If, as at , the p-value = 0.03, then the probability of randomly selecting a sample of 100 users who spend, on average, 25 minutes on the yellow-background website is only 3%!

In other words, the likelihood of getting a sample mean of 25 given that all sample means should be near 20 is only 3%.

So, in a world where the null hypothesis is true--a hypothetical world where the yellow background has no effect on the amount of time that people spend on the website, and thus, the mean amount of time spent on either website is 20--it would be very unlikely to get a sample of users from that hypothetical world who spend an average of 25 minutes on the yellow-background website. If that hypothetical world existed, then we would get a sample of users who spend 25 minutes on the yellow-background website only 3% of the time. If we continued to take samples of 100 users until we died, then we would of get a sample of users who spent 25 minutes on the new website 3% of the time.

So, the fact that this first sample returned a mean of 25 given that the null hypothesis is true is very unlikely--below the threshold that one assumes to be due to random chance alone--and therefore, the sample is inconsistent with the null hypothesis, and we reject it.
• How would I calculate the p-value if the problem doesn't give me a mean or standard deviation?
• What a confusing video
• Why do we calculate p(x>=observed value) and not just p(x=observed value)
• At , what do you mean by at least this far away from the mean, and why we wanna the P-value is the probability of X bar greater or equal to 25 mins?
• Because in step 3 at we're taking a sample and supposing that the sample mean is 25. So, we're asking a question: if we take a sample and the sample mean is 25 what's the likelihood of it happening given our population is 20? If the probability of getting a sample mean of at least 25 is very low (less than 0.05) then maybe the population mean is not 20 and we have reasons to reject the null hypotesis.
• Can someone please point me to the video where Sal explained how to calculate
P(x bar >= 25 minutes | Ha is true)?
Thank you
• I have a question on the procedure here. When the probability of ( X bar) >= 25 is below the significance level, why dont we question the sample the (X bar) came from instead of reject the null hypothesis. It may be that the sample we used might resulted in this unreasonable probability.
• Yes, that may be so, but you would probably choose a reasonable sample size, etc. while setting up the experiment and double check before calculating everything.
(1 vote)
• I am having a hard time wrapping my head around it, this is my understanding, please confirm if I am correct or correct me if I am wrong.

- So here we have 2 distributions, one is the distribution of the people who visit the website with white background and the other distribution is of the people who visit the website with yellow background.
We take sample data from the second distribution with a sample size of 100.

- p (x_bar >= 25 | H0 is true): is the probability of that the sample belongs to the first distribution or, in other words, probability that the sample was chosen completely by chance and not due to the fact that we changed the background.

- Alpha is the tolerance that is decided depending on the experiment.

- If the p-value is less than the tolerance, then we have reason to believe that the sample is different than the first distribution or it is very unlikely to happen by chance and more likely to happen because we changed the background color. And hence we have enough evidence to reject the null hypothesis.

- If the p-value is higher than alpha, then there is higher probability that the sample belongs to the first distribution or it is more likely to happen by chance and less likely to happen because we changed the background color.