If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

Main content

Introduction to experimental design

Introduction to experiment design. Creating a hypothesis. Double-blind testing. Placebo effect.

Want to join the conversation?

  • purple pi purple style avatar for user joychen04
    at Sal mentioned a "self fulfilling prophecy". What does he mean by this?
    (8 votes)
    Default Khan Academy avatar avatar for user
    • duskpin seedling style avatar for user Victor
      It just another way of explaining the placebo effect. That the people are making themselves faster based on a belief, but not the pill itself. It's "self fulfilling" belief basically and the effect is their prophecy if that makes sense. Hope it helps!
      (22 votes)
  • marcimus pink style avatar for user Jackie Miller
    I am going into 9th grade and this is really easy and basic .Is this the same as I was taught in 8th grade?
    (2 votes)
    Default Khan Academy avatar avatar for user
  • aqualine ultimate style avatar for user ★♪★♪★♪MontanaLis★♪★♪★♪
    Why does a hypothesis have to be testable in science?
    (4 votes)
    Default Khan Academy avatar avatar for user
    • duskpin ultimate style avatar for user Hugo Barbosa
      The scientific method is highly based on the empiric method, which is to say that there must be a way of verifying the truthfulness of what is proposed to explain certain phenomena. If there was no way for anyone other than the one who proposed the explanation in the first place to verify the hypothesis, it would not be very consistent because you would be either believing in the explanation because you regard who proposed it as a really smart and therefore trustable person or because you think the explication is good enough. But, as Sal said in this video, it is fundamental to verify the hypothesis by testing because if it is not correct, everyone who studies the phenomena in the future would be working on a false base (and it certainly would slow down the progress of science itself).
      (12 votes)
  • primosaur ultimate style avatar for user Mashiyat
    If I do the same experiment, but the groups are not random and they all have around the same average speed, would the experiment still be successful?
    (8 votes)
    Default Khan Academy avatar avatar for user
  • aqualine ultimate style avatar for user Michelle Mackenzie
    It's funny because science keeps improving, but we're taught the old stuff in school. :/
    (8 votes)
    Default Khan Academy avatar avatar for user
  • hopper cool style avatar for user GForceSquared
    At Sal talks about scientific progress can accumulate over hundreds of years; I was wondering if anyone can give an example of that?
    (6 votes)
    Default Khan Academy avatar avatar for user
    • spunky sam red style avatar for user Matthew Anderson
      I think when Sal says, "Scientific progress can accumulate over hundreds of years," he means that scientific method can always be built on. For example, in the 1500s, Copernicus proposed that the Earth orbits the sun (the common belief of the time was that the Earth was center of the universe. In the 1600s, Galileo published his findings that corroborated Copernicus' hypothesis. Fast forward several hundred years, as you know, we now accept this as common knowledge and have added vastly to our understanding of our solar system and theories of the universe.
      (5 votes)
  • blobby green style avatar for user hasoma25
    at when he talks about the placebo/control and the experiment. shouldn't there be a 3ed group in the experiment that didn't placebo or control pill at all that is just the normal average to compare to.
    (6 votes)
    Default Khan Academy avatar avatar for user
  • mr pants teal style avatar for user BarberAdam
    Why is the moon so tiny yet still is the same size as the sun from our distance?
    (4 votes)
    Default Khan Academy avatar avatar for user
  • blobby green style avatar for user Jaylen573
    does the hypothesis have to be tested and if not what action do you take
    (4 votes)
    Default Khan Academy avatar avatar for user
  • aqualine tree style avatar for user Priscilla Solorio
    What if the pill was not the reason they ran faster, what if it was something else how would I know it worked or not?
    (2 votes)
    Default Khan Academy avatar avatar for user
    • female robot amelia style avatar for user Johanna
      That’s why we do randomized, controlled experiments. For a controlled experiment, ideally all of the subjects would be exactly the same, so we could make sure that the only variable that changed among them was whether they got the pill or not. That way, we could only attribute changes in running speed to the pill. However, it’s not possible (or at the very least it would be highly unethical) to get humans to be exactly the same.

      To work around this, we do our best to control some variables (for instance by using a placebo so those in the control group also think they’re getting the flower pill). We also randomize who gets put in the control or experimental group to try to ensure that any significant difference between the groups is due to the pill.

      Taking these steps gets us closer to being able to say that it was in fact the pill causing the differences in speed.
      (3 votes)

Video transcript

- [Instructor] What we are going to do in this video is talk a little bit about experiments in science and experiments are really the heart of all scientific progress. If you think about, let's just say this represents just baseline knowledge and then people have hunches in the world and for a lot of times people say, hey, I have a hunch that that thing is good for you or that thing is good for you but they really had no way of measuring how confident they were. They really had no good way of proving it and even more, because they had no good way of proving it, it was hard for people to build on top of that knowledge but with the scientific method and experiments, people were able to say, hey, we have a hypothesis here and we were able to do some well-designed experiments and so we feel pretty good that this is true and then future people are going to say, hey, since we feel pretty good that this is true, maybe we can design experiment to see whether that is true. Hey, that actually is true and then they can build on that and we end up having scientific progress that can accumulate over hundreds of years and this is really important that the experiments are well-designed because in the future and this happens all the time, we might realize that hey, actually there was a little, a few assumptions baked in here that weren't accurate that allowed us to make essentially misleading conclusions. So our conclusion wasn't quite right there and then we will have to rebuild from that point in order to make sure that we are truly making progress. So the key question is how do we set up well-designed experiments and it's a whole field of study but the whole purpose of this video is to really give an introduction to it. So let's just start with a hypothesis. Let's say that you have a hypothesis that some pill that is made up of the petals of some flower, that this pill right over here, it improves, it improves running, running speed. It improves running speed if someone were to take it. So the important thing of any hypothesis, it has to be testable and so what you do is you have to think well, how am I going to test it? Well, what you can do, so how are you going to test your hypothesis? At first, you might say, give the pill, so give the pill to some runners, to some runners and test their time, test their 100 meter time, test their 100 meter time before the pill, before the pill and after and you might say, hey, maybe if, I don't know, their times improve after, maybe my hypothesis is correct. Pause this video and see if you feel comfortable with this test right over here, this experiment. Well, actually, there's several problems with this experiment. How are you selecting these runners and if you give them the pill and their speed improves, did it truly improve because of the pill or did it improve because of some other thing that they are doing? Maybe they got new shoes or maybe their diet improved in some way or maybe they just had a psychological improvement. This is often known as the placebo effect. If people are taking something that they think will help them, it often will help them even if that thing is just an empty capsule or just a sugar tablet. So how do you avoid these types of errors? Well, what you could do is you can find runners and put them into two groups. So let's say this is one group right over here and then this is another group and what you would wanna do is you'd wanna go into the population of people and you would want to randomly select whether someone goes into one group or another group. Why random? Because if you don't randomly select, there's a chance that there might be some implicit bias that you might just happen to be picking people who maybe their running speed is on an upward trajectory and they just happen to go into the group that will eventually get the pill. So you randomly, randomly put them in those groups and what you wanna do is you'll have a control group and you'll have a group that gets your pill and so this group gets the pill, gets the pill and now you might be tempted for this group to say, oh, they don't get a pill and then after a few months of it and it should be the same amount of time, you say, hey, did this group's times improve over the 100 meters? How did that compare to this group? But be very careful. If this group gets the pill and this group gets nothing then the pill might be providing that placebo effect again just making people think they're getting something that's making 'em faster. It might actually be a self-fulfilling prophecy. So it's actually important that you also give these people a pill although this pill would just look like a pill so this would be just an empty, empty, empty pill that looks the same. Now, there's another idea when you're designing scientific experiments that it needs to be double blind. Let me write this down. Double blind. So as you could imagine, it implies that two things are blind here. So the first thing that needs to be blind is the people themselves should not know which group they're getting put into. They should not know which pill they are taking because obviously, if you put someone in this pill, in this group and you say, hey, you're in the control group, we're just gonna give you an empty pill, well, then the placebo effect might not be, it might not work. It's also important 'cause it's double blind that the people who are working with the runners, the people who are measuring them so that the researcher is right over here so I'll draw someone with a clipboard. So the researchers who are observing these people and maybe administering the pill and telling them about the experiment that they too do not know which group they are administering it to because if they did, they might be able to signal somehow. They might be able to even subconsciously give a sense of which group folks are in and so let's say we do all of these things and so we're getting in the direction of a well-designed experiment and we find that the 10 people in this group versus the 10 people in this group after three months, these folks had a 5% improvement in running speed and these people had a 10% improvement in running speed. Is that enough to conclude that our hypothesis is correct? Well, you might be tempted, it seems suggestive but that's where statistics come into order because there's just some random chance that you got lucky, that you happened to pick the people that, your pill does nothing but you just happened to pick the people who are going to improve more and there's a whole field of inferential statistics when you take a statistics course that will go in more depth into this but essentially, what you're gonna do is you're gonna say, hey, assume that your pill does nothing, what's the probability of getting this result for 10 people or what's the probability of getting this difference in result and if that probability is very low, well, you say, hey, that would suggest that my pill actually does do something. Now, another important principle of an experiment like this is it needs to be replicable, replicable because even though you thought you did a good job, people might not wanna take your word for it and it's important in science for people to be skeptical. When people do experiments, they want to have a result and that bias might creep in and so if someone else does an experiment, you need to say how you did that experiment so other people can see if they get the same results because even though you think you randomly selected, you might only do it with people from a certain country or under certain weather conditions or assuming other constraints and then these people might do it slightly differently or in a different country or under different constraints and realize that hey, the explanation for this maybe was something else. Another thing to keep in mind is the larger your samples right over here, the larger groups that you're able to do this with, the stronger that the statistics actually become and I would say not just larger but the more diverse across genders, across ethnicities, across geographies. So the big picture here is all scientific progress is based on us designing good experiments and being very rigorous about how we think about those experiments and what I've highlighted here is just the beginning of how we might think about designing those experiments and as you go into your scientific careers, look at other people's experiments and see whether they've done these things because many times you will find that it is not as rigorous as it might seem at first.