If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

Main content
Current time:0:00Total duration:11:36

Video transcript

Giavanna usually takes Busbee to work but now she thinks that bus a gets her to work faster she randomized 50 workdays between a treatment group and a control group for each day from the treatment group she took bus a and for each day from the control group she took Busbee each day she timed the length of her drive and though this is really interesting what she did is very important she randomized the 50 workdays support she did this instead of just kind of waking up in the morning and just deciding on her own which bus to take because humans are infamously bad at being random even when we think we're being random we're actually not that random she might inadvertently be taking bus a earlier in the week where maybe the the commute times are shorter or maybe she inadvertently takes bus a when when the weather is better when there's less traffic and remember she you know there's a natural tendency for human beings to want to confirm their hypothesis so if she thinks that bus a is faster maybe she'll want to pick the days where it will she'll get data to confirm her hypothesis so it's really important that she randomized the 50 workdays and so what I could imagine she did is maybe she wrote each of the work day the dates on a piece of paper so she would have 50 pieces of paper and then she turned them all upside down and or maybe she closed her eyes and then she moved them all over her table and then with her eyes closed she randomly moved them to either the left or the right of the table and if they moved to the left of the table then those of the days she'll take plus a if she moves up to the right of the table those the day she takes bus B and that's how she can make sure that this is truly random all right so then they tell us the results this is important the results of the experiment show that the median travel duration for bus a is 8 minutes less than the median travel duration for bus B or one way to think about it if we said if we said the treatment treatment group the treatment group median minus minus the control group median control group median what would we get well the treatment group is 8 minutes less than the control group right this is a this is B so if this is 8 less than this then this is going to be equal to negative 8 this is just another way of reached eating what I have underlined right over here someone's car alarm went off hope your you're not hearing that anyway I'll try to pay attention while it's going off to test whether the results could be explained by random chance she created the table below which summarizes the results of 1,000 reran demise ations of the data with differences between medians rounded to the nearest 5 minutes so what is going on over here you might say well look she got her result that she wanted to get she sees this data seems to confirm that bus a gets her to work faster what's all this other business with rhe randomization she's doing well the important thing to realize is and she realizes this is that she might have just gotten this data that I underlined by random chance there's some chance maybe a and B are completely similar in terms of how long they take in reality and she just happened to pick bus a on days that are where bus a got to work faster maybe busing B is faster but she just happened to take bus a on the days that it was faster the days that just happened to have less traffic so what she's doing here is shiri randomized the data and she wants to see what what if with all this reran demise data out of these 1,000 re randomizations what fraction of them do I get a result like this do I get a result where a is 8 minutes or more faster I guess you or you could say that the median travel duration for bus a is 8 minutes less or even less than that then the median travel for bus B so if it was 9 minutes less or 10 minutes less or 15 minutes less those are all the interesting ones those are the ones that confirm our hypothesis that bus a gets to work faster so let's look at this table it's not below it's actually to the right so let's just let's just remind ourselves what she did here because the first time you you try to process this it can seem a little bit daunting so in a her experiment let me write this down experiment the car alarm outside which you probably hopefully are not hearing it's actually a surprisingly pleasant sounding car alarm sounds like a like a slightly obnoxious bird but anyway so her experiment is so the way I described it 25 days she would take bus a 25 days she took a bus B and she would record all the travel times and let's say that I just have 25 data points each column and so let's say they get 12 minutes 20 minutes 25 minutes and you just keep going there's 25 data points and let's just say that there are 12 data points less than 20 minutes and 12 data points more than 20 minutes and so in this circumstance or median time for bus a would be 20 minutes and I just made this number up and so in order for this to be 8 minutes less than the median time for bus B the median for bus B maybe we would have to be 28 and maybe you have data points here maybe this is 18 and you have 12 more that are less than 28 and then you have 12 12 more that are greater than 28 and so the median time for bus B the median time for bus B would be 28 once again I just made this data up and if you took contribute median and I'll just write I'll write TGM for short T gm- control group median control group median what do you get 20 - 20 8 is negative 8 this is this is the actual results of or these are theoretical or potential results hypothetical results for her actual experiment now what's all of this business over here well what she did is she took these times and she said you know what maybe let's just imagine a world where I could have gotten any of these times randomly on either bus so she just randomly Reese ordered them between she just randomly Reese ordered them between a and B and she did that a thousand times so the first time the second time the third time and she does this she does this one thousand one thousand times I'm assuming she's some type of a computer program to do it and each time once again she she just took the data that she had and she just rearranged it she just she just reshuffled it so maybe a on one day maybe it gets maybe it gets this maybe it got this 18 maybe it gets the 25 maybe gets a 30 once again so got the 18 the 25 of the 30 and maybe B gets the and you know she's reshuffling all this other data points that I just have with dots and maybe B let's see she had the 18 25 30 maybe 12 20 28 12 20 and 28 and so in this circumstance this random reshuffling and she keeps doing it over and over again in this random reshuffling the treatment group median minus the control group median is going to be what it's going to be equal to positive five in this random shuffling this hypothetical scenario bus A's median would have been five minutes more longer than bus B's and so if she gets this result with this random Reese orden this would have been and this is actually a this is a she would have had a column here four or five and then she would have she would have notched put one notch right over here but it looks like she she classified things or maybe she didn't even get the data but she classified them by multiples of two but then if she got this again then she would have put a two here and then she should have said okay and how many of these random reshuffling x' am I getting a scenario where there is a a five-minute difference so or where the treatment group is five minutes longer so what is this so what is this saying so for example for example this is saying that 18 out of the a thousand reshuffling x' which she just randomly reshuffled the data 18 out of those thousand times she found a scenario where her treatment group median was ten minutes longer than her control group where bus Ayotte is median was this hypothetical rear and emission where the treatment group is ten minutes slower than the control group there were 159 times where the treatment group once again in her random reshuffling they see these aren't based on observations these are random reefs off shufflings there's a hundred and fifty-nine times where her treatment group is four minutes slower than her control group so the whole reason for doing this is she says okay what's the probability of getting a result like this like this or better and I say better is you know I guess one that even more confirms our hypothesis that the treatment group is faster than the control group well the scenario this scenario is this one right over here and then another one that where the treatment group is this is even faster is this right over here here the treatment group median is 10 less is 10 less than the control group median so how many and how many of these scenarios out of the thousand is this occurring well this one occurs 85 times this one occurs 8 so if you add these two together 93 out of the thousand times out of Harare randomization or I guess you could say 9.3 percent of the time if the data if you night at nine point three percent of the randomized of the in the 1000 year and emissions nine point three percent of the time she got data that was as as validating of hypothesis or more than the actual experiment so one way to think about this is the probability of randomly getting the results from her experiment are or or better results from her experiment are nine point three percent so they're low it's a reasonably low probability that that this happened purely by chance now a question is is well what's the threshold if it was a 50 percent you say okay this was you know very likely to happen by chance if this was a 25 percent you're like okay it's it's less likely to happen by chance but it could happen 9.3 percent its roughly 10 percent you know every for every 10 people who do a an experiment like she did one person with even if it was random one person would get data like this so what typically happens among statisticians is they draw threshold and the threshold for statistical significance is usually five percent so one way to think about the probability of her getting this result by chance or this result or a more extreme result one that more confirms our hypothesis by chance is nine point three percent now if your if your cutoff for significance is five percent if you said okay this has to be five percent or less then you say okay this is not statistically significant there's a more than a five percent chance that I could have gotten this result purely through random chance now once again that's just depends on where you have that threshold so when we go back I think we've already answered the final question according to the simulations what is the probability of the treatment groups median being lower than the control groups media in by eight minutes or more which once again eight minutes or more that would be negative eight and negative ten and we just figured that out that was 93 out of the thousand re randomizations so it's a 9.3 percent chance and if you set 5% as your cutoff for statistical significance you'd say okay this doesn't quite meet my my my cutoff so I made maybe this is not a statistically significant result