Main content

## The idea of significance tests

Current time:0:00Total duration:4:41

# Comparing P-values to different significance levels

## Video transcript

- [Instructor] What we're
going to do in this video is talk about significance
levels which are denoted by the Greek letter alpha and we're gonna talk about two things, the different conclusions you might make based on the different significance levels that you might set and also
why it's important to set your significance levels ahead of time, before you conduct an experiment
and calculate the p-values, for, frankly, ethical purposes. So to help us get this,
let's look at a scenario right over here which tells us Rahim heard that spinning,
rather than flipping, a penny raises the probability above 50% that the penny lands showing heads. That's actually quite
fascinating if that's true. He tested this by spinning
10 different pennies 10 times each, so that would
be a total of a hundred spins. His hypotheses were, his
null hypothesis is that by spinning, your
proportion doesn't change rather versus flipping, it's still 50% and his alternative hypothesis
is that by spinning, your proportion of heads
is greater than 50%, where p is the true proportion of spins that a penny would land showing heads. In his 100 spins, the penny landed showing heads in 59 spins. Rahim calculated that the statistic, so this is the sample proportion here, it's 59 out of a hundred
were heads so that's 0.59 or 59 hundredths, and he calculated had an associated p-value
of approximately 0.036. So based on this scenario,
if ahead of time, Rahim had set his
significance level at 0.05, what conclusions would he now make? And while you're pausing
it, think about how that may or may not have been different if he set his significance
levels ahead of time at 0.01. Pause the video and
try to figure that out. So let's first of all remind ourselves what a p-value even is. You could view it as the probability of getting a sample
proportion at least this large if you assume that the
null hypothesis is true. And if that is low enough, if it's below some threshold, which is our significance level, then we will reject the null hypothesis. And so in this scenario, we do see that 0.036, our p-value
is indeed less than alpha. It is indeed less than 0.05 and because of that, we would
reject the null hypothesis. And in everyday language,
rejecting the null hypothesis is rejecting the notion that
the true proportion of spins that a penny would land
showing heads is 50%. And if you reject your null hypothesis, you could also say that suggests
our alternative hypothesis that the true proportion of spins that a penny would land showing
heads is greater than 50%. Now what about the situation where our significance level was lower? Well in this situation, our p-value, our probability of getting
that sample statistic if we assumed our null
hypothesis were true, in this situation, it's
greater than or equal to, and it's greater than in
this particular situation, than our threshold, than
our significance level. And so here, we would say that we fail to reject our null hypothesis
so we're failing to reject this right over here and
it will not help us suggest our alternative hypothesis. And so because of the difference between what you would conclude given this change in significance levels, that's why it's really important to set these levels ahead of time because you could imagine
it's human nature, if you're a researcher of some kind, you want to have an interesting result. You want to discover something, you want to be able to tell your friends, hey, my alternative hypothesis
it actually is suggested, we can reject the
assumption, the status quo. I found something that
actually makes a difference and so it's very tempting for a researcher to calculate your p-values and then say oh, well maybe no one will notice if I then set my significance values so that it's just high enough so that I can reject my null hypothesis. If you did that, that
would be very unethical. In future videos, we'll start
thinking about the question of okay, if I'm doing it ahead of time, if I'm setting my significance
level ahead of time, how do I decide to set the threshold? When should it be one-hundredths? When should it be five-hundredths? When should it be 10-hundredths? Or when should it be something else?