Ways to measure impact

- [Voiceover] What I want to do in this video is talk about the different ways that, as an organization, we can measure our impact on the world. This is really the return that a not-for-profit cares about, we aren't here to generate profit for shareholders, we're here to generate impact for all of our stakeholders, from our donors, for our own team, all of us investing a lot of time in this, to, most importantly, the learners of the world. The first layer that we can think about, and this, in some ways, you can think of as the easiest to measure, is our site metrics. Our site metrics. And there's many, many metrics we can look at, and when we do A/B tests, we can measure how it's affecting different things, but the one that encapsulates the high level, how much impact we're having in the world, is total learning time. Total learning time, which we talk about in other videos, is the product of our reach and our engagement. Now, we will, of course, look at other types of metrics, but one really has on aggregate, how much are we being used by the world, given the assumption that usage correlates with actual learning, with actual people tapping into their potential. Now, that last sentence I just said, that assumes that every minute on Khan Academy leads to actually learning, actual learning, but you can imagine, there's ways of maximizing total learning time, where it's not about learning, where it's about entertainment or it's about something else, and so, we do want to look at other metrics that make sure that we don't lose our way as we just try to optimize total learning time. The next layer that we can think of beyond just our direct site metrics, and site metrics are going to be really important, we're going to talk a lot about these over the next 10 years because they're measurable, you can A/B test around it, it's very easy to rally around these things, but beyond the site metrics, the next layer in terms of how easy it is to measure, and it gives us a little bit more texture, more than just sheer learning time, or time on the site, is, I guess you could call them, survey metrics. Survey metrics. And actually, let me split this up. Let's first say you have site surveys. Site surveys, so these are things you can ask our actual users, so these are things like net promoter scores or peoples' assessment of quality of certain pieces of content. We could also use that to understand the usecases, asking students, "Well, why are you here at Khan Academy? "Are you finding it useful? "Would you promote this to a friend?" And we can do this pretty regularly, and pretty automated, so it's close to site metrics in how measurable it is, but it's not quite as precise as the site metrics. Now, the next layer is market surveys. Market surveys. Obviously, site surveys are just looking at the subset of the world that happens to go to the site, and happens to fill out the surveys. Market surveys you could look broader, and, as we talk about it in our goals in terms of impact, we want to have an impact that is noticeable from a market perspective. And so, these are things like the survey, "Has Khan Academy had a meaningful impact "on your education?" We've already done a round of that at some selective colleges, and we were surprised by how high the number was, places like Stanford, and Harvard, and MIT, we saw over 60% of students say that Khan Academy has had a meaningful impact on their education. Amongst the students, the subset, who were the first in their family to go to college, it was 64% had a meaningful impact on their education. So, this is something we want to focus on as we go forward, as we try to execute on this vision, we want to make sure it's not just at selective universities even though we did look at the subset of first generation students, we want to look at that more broadly, and also, internationally. "Has Khan Academy had a meaningful impact "on students' education?" And then, we can ask them, "How has Khan Academy "had a meaningful impact on your education?" We have some theories, but we might get surprised by the different ways that students will say that Khan Academy has been able to help them. There's other things that we could do. I really like this first one, but we can do things like brand surveys. Brand surveys, just in terms of awareness, and how people perceive us, and things like that. Then, there's things, and this was pretty exciting, things that we've been able to do like the college board, around the SAT, where we were able to survey takers of the SAT, and we saw over half of all SAT takers used Khan Academy. 98% of those found it was useful or very useful. We already saw 20% reduction in paid test prep. We saw usage of the SAT, it was broad across ethnic and economic demographics, so these are really, really important for making sure that we stay true to the free world-class education for anyone, anywhere, and don't somehow forget our values and only optimize learning time. Now, if you want to get to a level deeper than even market surveys, you can do efficacy studies. Efficacy studies. And we've already had some efficacy studies done. We had the SRI study that got completed a few years ago, We had the Idaho study. Both of these were able to correlate Khan Academy mission use to better-than-expected test scores on the NWEA MAP exam. That was pretty powerful, but it's also the case that these are hard to do. It's hard to do control studies in classrooms, where there's so many variables, and Khan Academy is only a part, different levels of fidelity of a part of a classroom experience. This is pretty hard, and that's why I'm also excited about what we're going to be able to do with the PSAT. PSAT, and there's obviously a PSAT in 10th grade, but there's going to be one in the eighth grade, possibly sixth grade, the PSAT/SAT, because this is really a nationwide ability for us to say, "OK, how did someone do on the pre-test? "What was their engagement on Khan Academy? "What type of activities did they do, and then how did that impact their college readiness?" And it could be around exams like this, as we go into other subjects, we can look at benchmark exams like AP exams, CLEP, we can look at Accuplacer, and how effective was Khan Academy as students actually used it? So, over time, once again, we can come up with numbers here almost in real time, here, we can get numbers, it takes a little bit more time, we have to be a little bit more careful in terms how we phrase the questions, in terms of who sees it, in terms of how we interpret it. That's, frankly, true of all of this. Efficacy studies, these SRI, Idaho, classroom-type studies we've already done a few around missions, and we were able to do them once every two years or so, but what I'm excited about the SAT/PSAT is this might be able to be a much faster, and much more precise, and have higher fidelity, and we're going to be able to look more longitudinally over students as they go through life, from sixth grade, to eighth grade, to 10th grade, to college. Hopefully, maybe even see them as they go through college. And then, the last layer, and this is last but not least, are the testimonials. Testimonials. In a lot of ways, testimonials, they aren't really numbers-driven, but they tell a very important narrative to us of how Khan Academy is being used, and how, in particular, it's changing individual students' lives. We just got a testimonial from a student in Afghanistan, who was threatened with acid attacks when she was in middle school, and then she got her family to give her an internet connection. She's middle-class, so this was a significant fraction of their income. She was able to learn English through the internet and newspapers, and she's self-educated on Khan Academy, and now, she had to be smuggled into Pakistan to take the SATs, and now she's coming, or she hopes to come to the US to study physics. This is the type of thing that just can't be captured in any of these other numbers, but is an important dimension to the vision of Khan Academy, that we can reach these students who have nothing else. And even, as individuals, can help push all of our society forward, and that was just a testimonial that we got last week, we have stories of Zai, an orphan in Mongolia who then became one of the top contributors to content. And we get tens or hundreds of these, almost on a daily basis, that aren't quite that powerful, but speak to people, tap into their potential, people going back to college, people with a learning disability, people who thought that they weren't capable, so it's really important that we never lose sight of this as well, as we look at all of the numbers.