Scientific polling introduction

- [Instructor] In this video, we're gonna think about what makes a poll or a survey credible. Because remember, the whole reason why we're going to do a poll or a survey is we wanna understand public opinion. But if it's not statistically credible, if we can't believe what it's saying or if we don't understand exactly what it's saying then we might not have a good sense of what public opinion actually is. So the first really important thing, if you're writing any type of survey, whether it's in politics or government or not, is that you are taking a random, random, sample. What does that mean? Well we go into a lot more depth in it in our statistics content on Khan Academy, but it means that you should look at the people, the voting population. And it does get a little bit nuanced and tricky on who are likely voters, but you would go to say the voting population or the people in an area whose opinion you care about, and you would want any one of them to have an equal shot of being selected for your survey. So good examples of random samples is maybe you take, you have a random phone number generator of likely voters in your district and you call them up. Maybe, and even then you might say well hey, maybe certain types of people pick up the phone and certain people don't, or certain people have a phone and certain people don't. So you have to be very careful about this design. But examples of non-random sample would be to go hang outside of the Democratic party headquarters in your district and just survey people coming out of that building, that would be very not random and you would get very skewed results. Now, the next thing that you would want when you are taking a survey is that you want a large population size. Now, typically speaking, it's going to be at least 500 folks, but you're going to see a lot of surveys that are about 500 to 1,000 participants. And sometimes in statistics they'll say your N, which is the number of people you surveyed is 500 to 1,000. And this is so you have a good chance of getting close to the true public opinion. We're gonna talk in a second about margin of error. The larger this is, and especially if you're doing a true random sample, this is so, larger, larger, it reduces the, you'll have a lower, margin, margin, of error. And I'll explain in depth what a margin of error is in a second. Now, another key thing is whatever you ask in your survey, it needs to be, the language, needs to be as neutral and unbiased as possible. Let's say there's a new proposition, you could get very different results if the wording of the question is do you support funds for those in need or if you said do you support funds for those who are not working. Those could get very, very different results. And this is, there's really an art to trying to get a neutral, unbiased question there. Last, but not least, you wanna make it very transparent to the public how you conducted your poll so that they can decide for themselves how credible your poll actually is. So with that out of the way, let's look at real poll results. And then I'm gonna dig into what the idea of margin of error is, because you'll see this a lot when you read the news. So this is from the Sun Sentinel in Florida. They have a senatorial race going on, and it says despite millions of dollars in television ad spending, Florida's US Senate race between Bill Nelson and Rick Scott hasn't budged. A Florida Atlantic University poll shows Republican Scott with 44% of voters surveyed, Democrat Nelson with 40%. So the first thing you might say is well, what about this only adds up to 84%, what about the other 16%? Well, those could be undecided voters or maybe there's a third candidate there. That's a four-point advantage in Scott's favor, but it's within the survey's margin of error, which means the race could be tied, or Scott could have a lead. The FAU Business and Economics Polling Initiative survey of 800 Florida registered voters was conducted online and through automated calls to people with landline telephones. Researchers said it had a margin of error of plus or minus three percentage points. So let's think about whether this is a credible poll and what inferences we can make from it. So the first thing is, were they transparent about their methods? And it seems like they were, this whole, this paragraph right over here, they're very transparent about their methods, how many people did they survey, they say how they actually conducted the survey, and so check on transparency. And then because they were transparent, we can check whether these other things are true. So first, random sample. They don't go into a lot of depth on how it was conducted online, but if it is a neutral site where any Florida voter is equally likely to show up, well then that might be a good random sample. Now, you have to be careful with online because who has access to online, who does not have access to online, who might go to whatever site it is being conducted on. So this has a couple of question marks right over here. And then they say through automated calls to people with landline telephones. So once again, this seems reasonably random, it's along the lines that I talked about earlier. Maybe they had some random phone number picker in Florida and they called those folks, but maybe, you know, there's certain people have landlines, a lot of people now only have mobile phones. There's certain people who pick up and might answer things and certain people who might not. So as you can see, there's an art to getting a truly random sample, but you can tell that the Florida Atlantic University group tried to get a random sample. The next was the large population size. Well, they said a survey of 800 Florida voters, so that's pretty good. And it's not just 800 for 800 sake, it's the number of voters that's gonna drive the margin of error. So this is a good time to say what is the margin of error? So a margin of error is going to be associated with a confidence level. It's typically going to be a 95% confidence level if they don't tell you otherwise. And so what this means is, is that 95% of the time, when you take a random sample of 800 Florida residents, you will get a result that is within three percentage points of the true result. Remember, there's some true result we don't know unless we could perfectly get into everyone's head and this poll is a way of trying to estimate that. So one way to think about it, within 3% of what you got, Republican Scott is really, you could create a confidence interval where you could say the confidence interval's going to be 3% less than this, which is 41%, all the way to 3% more than this, which is 47%. And one way to think about it is, there's a 95% chance that this will overlap with the true result. And you might say hey, well the low end of this range is still higher than Democrat Nelson, but that confidence level applies to Democrat Nelson as well. So Democrat Nelson's confidence interval would be between 37%, 3% less than this, and 43%, 3% more than this. There's still a reasonable chance maybe the Democrat's at the higher end of this range which could still mean that the Democrat could win even though the headline numbers show a four-point advantage for the Republican. And outside of the statistical error, the margin of error we're talking about, you have to remember that this poll is before the actual election. People change their minds, there's a whole campaign going on, so even if you were to get the exact result, this is just a snapshot in time. It could change on election day. Now this last question where they asked a neutral, unbiased question, we don't know exactly, at least just from this article, they might have published it in the details of the poll, but if they asked something like who will you vote for on the election on this day, that would be very neutral and unbiased. But if they say are you going to vote for the public servant who has served Florida for many years or that Democrat guy, well that would be not unbiased.