Introduction to experiment design

- [Instructor] So let's say that I am a drug company and I have come up with a medicine that I think will help folks with diabetes, and in particular, I think it will help reduce their hemoglobin A1c levels, and for those of you who aren't familiar with what hemoglobin A1c is, I encourage you, we have a video on that on Khan Academy, but the general idea is if you have high blood sugar over roughly a three-month period of time, high blood sugar, and I can say high average blood sugar, you're going to have a high A1c, a high hemoglobin A1c level and if you have a low average blood sugar over roughly a three-month time, you're going to have a lower hemoglobin A1c. So if taking the pill seems to lower folks' A1c levels more than is likely to happen due to randomly or due to other variables, well then that means that your new pill might be effective at controlling folks' diabetes. So in this situation, when we're constructing an experiment to test this, we would say that whether or not you are taking the pill, this is the explanatory variable. Explanatory variable, and the thing that it is affecting, the thing that you're hoping has some response, in this case the A1c levels are your indicator of whether it is help controlling the blood sugar, we call that the response variable. That right over there is the response variable. So how are we actually going to conduct this experiment? Well let's say that we have a group of folks, let's say that we have been given a group of 100 folks who need to control their diabetes. So 100 people here who need to control their diabetes, and we say, "Alright, well let's take half of this group "and put them into, I guess you could say a treatment group "and another half and put them into a control group "and see if the treatment group, the one that actually "gets my pill is going to improve their A1c levels in a way "that seems like it would not be just random chance." So let's do that, so we're going to have a control group, so this is my control group, control, and this is the treatment group, this is the treatment group. And you might say, "Okay, we'll just give these folks, "the treatment group the pill and then we won't give "the pill that I created to the control group." But that might introduce a psychological aspect that maybe the benefit of the pill is just people feeling, "Hey I'm taking something that'll control my diabetes," maybe that psychologically affects their blood sugar in some way and this is actually possible, maybe it makes them act healthier in certain ways, maybe that makes them act unhealthier in certain ways 'cause they're like, "Oh I have a pill to control "my diabetes, my blood sugar, I can go eat "more sweets now and it'll control it." And so to avoid that, in order for just the very fact that someone says, "Hey I think I'm taking a medicine, "I might behave in a different way or it might even "psychologically affect my body in a certain way," what we wanna do is give both groups a pill, and we wanna do it in a way that neither group knows which pill they're getting. So what we would do here is we would give this group a placebo, a placebo, and this group would actually get the medicine, the medicine, but those pills should look the same, and people should not know which group they are in and that is a, when we do that, that is a blind experiment, experiment. Now you might have heard about double-blind experiments. Well that would be the case where not only do people not know which group they're in, but even their physician or the person who's administering the experiment, they don't know which one they're giving, they don't know if they're giving the placebo or the actual medicine to the group. So let's say we wanna do that. So we could do double, double-blind experiment, so even the person giving the pill doesn't know which pill they're giving. And you might say, "Well why is that important?" Well if the physician knows, or the person administering or interfacing with the patient, they might give a tell somehow, they might not put as much emphasis on the importance of taking the pill if it's a placebo, they might by accident give away some type of information. So to avoid that type of thing happening, you could do a double-blind, and there's even, some people talk about a triple-blind experiment where even the people analyzing the data don't know which group was the control group and which group was the treatment group, and once again, that's another way to avoid bias. So now that we've kinda figured out, we have a control group, we have a treatment group, we're using A1c as our response variable, so we would wanna measure folks' A1c levels, their hemoglobin A1c levels before they get either the placebo or the medicine and then maybe after three months, we would measure their A1c after, but the next question is, how do you divvy these 100 people up into these two groups, and you might say, "Well I would wanna do it randomly," and you would be right 'cause if you didn't do it randomly, if you put all the men here and all the women here, well that might, first of all, sex might explain it or behavior of men versus women might explain the differences or the non-differences you see in A1c level, if you get a lot of people of one age or one part of the country or one type of dietary habit, you don't want that, so in order to avoid having an imbalance of some of those lurking variables, you would want to randomly sample and we've done multiple videos already on ways to randomly sample, so you're going to randomly sample and put people into either groups. And a very simple way of doing that, you could give everyone here a number from one to 100, use a random number generator to do that and then, or you could use a random number generator, pick 50 names to put in the control group or 50 names to put in the treatment group and then everyone else gets put in the other group. Now, to avoid a situation, just randomly by doing a random sample, you might have a situation where there's some probability that you disproportionately have more men in one group or more women in another group and to avoid that, you could do really a version of stratified sampling that we've talked about in other videos, which is you could do what's called a block design for your random assignment where you actually split everyone into men and women and it might be 50-50 or it might even be just randomly here you got 60 women, 60 women and 40 men and what you do here is you say, "Okay let's randomly "take 30 of these women and put 'em in the control group "and 30 of the women and put 'em in the treatment group "and let's put randomly 20 of the men in the control group "and 20 of the men in the treatment group" and that way someone's sex is less likely to introduce bias into what actually happens here, so once again, doing this is called a block design, really a version of stratified sampling. Block design, and there might be other lurking variables that you wanna make sure doesn't just show up here randomly and so you might want, there's other ways of randomly assigning. Now once you do this, you see what was a change in A1c. If you see that, hey, the change in A1c, one if you see there's no difference in A1c levels between these two groups, and you're like, "Hey, there's a good probability that my pill does nothing" and once again it's all about probabilities, there's some chance that you're just unlucky and it might be a very small chance and that's why you wanna do this with a good number of people and as we forward our statistics understandings, we will better understand at what threshold levels do we think the probability is high or low enough for us to really feel good about our findings. But let's say that you do see an improvement, you need to think about, is that improvement, could that have happened due to random chance or is it very unlikely that that happened due purely to random chance, and if it was very unlikely that it happened due purely to random chance, then you would feel pretty good, and other people when you publish the results, would feel pretty good about your medicine. Now, even then, science is not done. No one will say that they are 100% sure that your medicine is good, there still might have been some lurking variables that we did not, that our experiment did not properly adjust for, that just when we even did this block design, we might have disproportionately gotten randomly older people in one of the groups or the other or people from one part of the country in one group or another so there's always things to think about and the most important thing to think about, even if you did this as good as you could, you still, some random chance might have given you a false positive, you got good results even though it was random, or a false negative, you got bad results even though it was actually random. And so a very important idea in experiments and this is in science in general is that this experiment, you should document it well and it should be, the process of replication, other people should be able to replicate this experiment and hopefully get consistent results so it's not just about the results, it's your experiment design, other people should, it should be an experiment that other people could and should replicate to reinforce the idea that your results are actually true and not just random or just due to some bad administration of the actual experiment.