Impact from using Khan Academy

by Shantanu Sinha

We received an email a couple of months ago from a young man named Mark Halberstadt.  In true Khan Academy style, he recorded a YouTube video for us to watch. 

His story is quite inspirational.  Mark is a student who had given up on Math and Science and thought he was incapable of ever pursuing a career in a related field.  He claims he was always a “C” student growing up and never had a channel to understand topics that interested him in engineering.  He found the Khan Academy in 2007 and started watching videos on Trigonometry, Calculus, and even Arithmetic.  He decided to go back to school last year to get a Bachelors degree in Electrical Engineering.  He finished his first year in college with a 4.0 GPA for the entire year, including perfect scores on his Calculus and Chemistry final exams.  He says, “coming from a background where my GPA graduating from high school was in the 2.0 range.   That never would have happened – getting a 4.0 GPA would never have happened without the help I got from the Khan Academy.”  He goes on to say, “It has helped me immensely.  The impact for me in my life…I see it growing exponentially over the next 20 or 30 years." 

Students have always liked Khan Academy.  The YouTube comments Sal received when he posted his first videos in 2006 are what motivated him to keep going (creating over 2400 as of today).  However, it is still astonishing to see the impact our resources can have.  Of course, while Mark is thanking us, he deserves 100% of the credit.  He did all the work and took control of his own learning after the educational system left him behind.  We just played a small role in enabling him.

If there were only 1 student like Mark Halberstadt in the world, it would have been worth creating Khan Academy.  The fact that this could so easily scale to millions is what makes Khan Academy special.

Khan Academy seems to work well with supplemental learners like Mark, but how well does it work in schools?  We have always believed that a great teacher can take our resources and push learning to new heights, by better focusing on the individual needs of each student.   With the student mastering core skills on the computer, the teacher can leverage the classroom time for more engaging and dynamic activities such as project-based learning, peer tutoring, or lively discussion.

Last school year, we started piloting our platform in a few schools in Los Altos, California.  Our goal was to create better tools by directly observing how teachers and students interacted with our product.   Los Altos was a fantastic partner, and our team built out many significant features based on their feedback (e.g., student knowledge map, teacher dashboards, badging infrastructure, new exercises).   

At the end of the school year, we all knew it was a success.  Teachers could see a dramatic change in their students’ excitement and enthusiasm towards Math.  Students who traditionally struggled with the material were more confident and engaged.  Other students were challenging themselves to levels we never thought possible.  Common sense told everyone involved that we were on to something.   

We did not do a controlled research study.  In part, because our organization was only 5 people for most of the school year, and we were just trying to build something worth researching.  Things changed fast for us, and the system the students were using at the end of the school year was very different from the system they started using in November.

However, we were curious to see how they did on traditional assessments like the end of year CST exam.  It is not the ideal exam since it only tests performance on a narrow set of grade-level skills.   Many of our students were remediating topics they should have learned years ago, or challenging themselves with much more advanced topics.  None of these gains would be captured.   However, the CST clearly matters, so it is worth understanding how our students performed.

The initial results were quite promising.  Our pilot included a couple of 7th grade classrooms with students who typically struggled in Math.    We saw a significant improvement in this group.  The number of Advanced or Proficient students increased dramatically, from 23% to 41%

This was very heartening.  Usually, the performance gap widens with students who struggle in Math, particularly when they get to more advanced topics like Pre-algebra.  The fact that these students were closing the gap (non-pilot classrooms saw no significant change in their CST performance) was very promising. 

Our pilot also included a few 5th grade classrooms.  Los Altos is a high performing district, and these students typically do very well on the CST.  This year was no different, with 96% of the students in pilot classrooms scoring Advanced or Proficient.   While these are great results, they are not statistically different from the non-pilot classrooms in the district.    It turns out, in both pilot and non-pilot classrooms, the students were doing great on the exam and didn’t have much room for improvement.

However, we could see amazing things happening with the 5th graders.  A majority of students were attempting early Algebra, and many students were experimenting with Trigonometry and Calculus.  These students were excited, engaged, and loved being challenged.  Inadvertently, we highlighted a distinct but not often discussed problem with standardized, age-focused education.   Students performing at high levels are often not sufficiently challenged.  Teachers shouldn’t take kids who already know the material, and make sure they already know the material.  Teachers should be pushing and challenging the students to their full abilities.  Los Altos didn’t think everything was perfect because their students were scoring well on standardized exams; they saw significant value in creating an environment that was engaging and challenging for all students.

Based on these experiences, Los Altos has now decided to expand the implementation district-wide to over 40 more classrooms.  We are also working with a number of additional schools that represent different use cases (e.g., charter, independent, low-income, special needs) to understand how students react to our resources in these different settings.  This year we will also look into a better evaluation methodology that reflects learning gains across multiple grade levels.

We are completely convinced that our resources can have a huge impact on the learning process.  But why exactly does Khan Academy work?   Some people have a hard time understanding how online videos and practice exercises can make such a big difference.  Or they misunderstand what Khan Academy is all about.    

Putting videos on YouTube is just a small piece of the equation.   What Khan Academy enables is a fundamentally different way for students to approach learning.   Here’s my take on the many innovations we are bundling together into a coherent experience. This is what is really making the difference.

  • Students are free to learn anytime, anywhere
  • Students can jump to where help is needed most, and spend as much time as necessary to master concepts 
  • The content is short, fun, approachable, and easily digestible
  • There is a clear and continuous path to learning complex topics
  • Students feel an increased sense of ownership - they are learning, not "being taught”
  • The focus on core conceptual understanding ensures students build the necessary skills that are applicable in any curriculum used in schools
  • Interactive practice ensures concepts truly sink in
  • Rich data helps teachers monitor progress and provide focused support
  • Teachers are empowered to make their classroom experiences much more fun, engaging, and social, with less lecturing and more project-based learning and peer tutoring

We learned a lot this past year, and I suspect we will learn much more this coming school year.   The results so far have been promising.   However, in our view, we’re just getting started.  We got office space and started building a team only 10 months ago. We still have a long way to go to reach our vision for technology-enabled education.  


Bringing Creativity to Class Time (by Sal Khan)

I recently gave a talk organized by Intel to computer science and engineering professors at some of the top universities in the country (including my own alma mater).  I was asked by one of the professors how I would use something like Khan Academy in the context of an engineering education.  I told him that I’d make projects the primary mode of learning (as opposed to something that just complements lectures and exercises).  Khan-like  lectures and problem sets would then be used as on-demand tools to facilitate the projects and free class and professor time from lecturing about core concepts.  Not only would it provide for deeper, more motivated learning, but a portfolio of meaningful projects is vastly more interesting for a prospective employer than a GPA on a transcript (we know because Khan Academy’s own hiring process cares a lot more about what someone has creatively produced than their GPA).   I also added that this is how I think ALL education should be, not just engineering and not just higher education (projects can be anything from composing music to writing software).

I explicitly remember working on a project my fourth year in college trying to relearn a good chunk of signal processing that I had first been exposed to in a class two years prior.  I wished that I had those lectures on-demand in granular chunks so that I could fill in my gaps.  I wished that I could have feedback on some exercises to make sure that I had the basics down solid enough to not do something disastrous in the image processing library I was writing.  I wished that I had the ability to interact with the professor that I knew two years prior only as a lecturer and that only knew me as a face in crowd.  Given this, wouldn’t it have been 10 times better if my focus during that signal processing class 2 years ago was this project?  Wouldn’t it have been even better if I had lectures and exercises on-demand when I was ready for them.  Wouldn’t it have been great if the professor’s time (and my peers’ time) were liberated to advise and mentor me in my project instead of lecture?  Wouldn’t a solid piece of image processing software be a more impressive product of a signal processing class than an “A-”? Wouldn’t a portfolio of substantive and deep projects be more meaningful than a traditional transcript?

There is  a reality that we live in a world of tests that, for better or for worse, society uses to measure the competence of students and sometimes teachers and schools.  These are used to filter people in and out of schools and careers and can dictate the direction of a student’s future.  Some of these do have some merit, but, because there are such high stakes involved, preparation for the skills deemed necessary for these exams crowd out creative activity during scarce class time.  To completely ignore this testing reality does a disservice to students, but to cater 100% to it would be equally damaging.  We think the Khan Academy offers a third way.  Namely, it can be used to allow the core skills develop at a student’s pace and only during a fraction of class time.  This liberates the rest of class time for peer tutoring, higher level interactions between teachers and students, and truly creative projects.

I want to be clear how deeply rooted this is in our core philosophy.  I’ve co-run two summer camps where the students did exactly this–they spent 80-90% of their time building robots, experimenting with truly novel questions, and mentoring each other.  This was done in 2009 and 2010 before Khan Academy got its first funding and before I thought that this could be the focus of my life (I used the bulk of my vacation time from my day job at the time to run those camps).  The Khan Academy videos and software at the time (they were very primitive then) freed me from lecturing or administering problem sets and allowed me to dive deep into the students’ projects (including their misconceptions on the core material).  this is what the camps looked like:

Some of the other projects we did included:

  • Having the students estimate the value of coins in a a large jar given any tool at their disposal (including sample coins, scales, rulers, spreadsheets).  We then took pictures and posted them on Mechanical Turk to see if the “crowd” estimate was better (we payed 0.01 per guess) even though their information was not as good.  This led to a fascinating discussion of information and noise and when crowd estimates could be better than experts. 
  • Six students played a variation of Risk called “Paranoia” Risk where every player had the secret mission to eliminate one other player from the board (you only knew who you had to eliminate; you had to try to figure out who was charged with eliminating you). Once a player is eliminated, the winner is declared and the game is over.  All of the other non-playing students were each given $500 in monopoly money and a colored strip of paper representing each of the players on the board.  At the end of the game, a colored strip is worth 100 if that player won and 0 if they lost.  The students then traded slips as the game progressed.  Several students independently developed spreadsheet models based on the probabilities involved in the game.  Led to a deep discussion around information in markets and when bubble behavior develops (for example, several securities irrationally traded above 100) .
  • We played a variation of freeze tag where we changed the size of the playing field and the number of “freezers”.  Students predicted and observed how the dynamics of the game changed as more freezers were added and the threshold needed to freeze everyone.  They were quick to draw analogies to other areas of science.

From the energy, competence and creativity that I saw naturally coming from these students, it convinced me that this is how all school time should be.

Perhaps the two most difficult aspects of fully transitioning to a project and exploration based model is the fixation on high-stakes tests and the energy and coordination needed for truly deep explorations (not cook-book pretend science/engineering).  We think the current incarnation of the Khan Academy can help the former (we are already seeing a dramatic transition to project-based learning in every Khan Academy formal pilot), but we are just in the top of the first inning (we got our office space 9 months ago).  Over time, we will include deeper simulations, modules and tools for deeper explorations.  We will also try to integrate suggestions for deep projects based on proficiency in core skills.  We are even considering starting a physical school here in Silicon Valley that would be a testing ground for the cutting edge in student exploration and creative activity.  Here is a direct quote from an email I sent to a potential supporter about how we envision this school:

“I’d have students focused on building, creating and exploring real-world projects– from composing original music to building robots, writing mobile applications or running truly novel experiments.  The Khan Academy software and videos would facilitate this by allowing many of the core skills to be developed independently and it would be a resource when the students hits a need to develop a core skill because of their projects (I now *need* to learn trigonometry and matrices to figure out how this graphical object in my mobile app will look when it is rotated).  As we fine tune the projects and explorations that work in this setting, we will attempt to integrate them more deeply into the core Khan Academy platform so students and teachers around the world have the infrastructure and tools to fully explore their creativity.”


Next wave of school pilots

We are looking for a small set of schools to partner with in the Fall of 2011, for a hands-on Khan Academy blended-learning implementation.  These pilots will be focused on Math, and will include classroom visits and direct collaboration with our team.  Since we have a very small team (only 8 of us right now), we unfortunately must limit the number of pilots.  As always, Khan Academy’s online resources are completely free for any school to use on their own, independent of our team’s involvement.

If you would like to nominate a school, please contact the school and ask the appropriate decision maker (e.g., superintendent, principal) to complete this survey.  We will use the information to track interest and coordinate with the school.

Apply to become a pilot school

Applications must be submitted by midnight (Pacific time) on Tuesday, May 31, 2011.  Selected pilot schools will be notified by Friday, July 15, 2011.

In case you missed it, you can learn more about Khan Academy in the classroom by watching these videos posted on Bill Gates’ blog.


The Right Way to Use Technology in the Classroom

Originally posted on Shantanu’s Huffington Post blog (3/10/11) 

There have been many attempts to incorporate technology in the classroom. Since the early ‘80s, schools have been stocking labs with the latest gadgets. Clearly, the world was changing.  We’ve seen the rise of personal computers, the internet, mobile devices, streaming video, social networking, and tablets. Our educational system must be able to benefit from these advances.

However, most attempts have been fundamentally flawed.  Computers were installed with mildly educational games that were kind of “cute.” Teachers weren’t provided the right tools to properly integrate the technology into their day-to-day instruction. For the most part, we didn’t teach kids with the computer, we taught them how to use the computer. Most kids need no help and could probably teach their parents. In the end, computer labs were a side show, expensive investments largely squandered due to a lack of good content or purpose.

We at the Khan Academy have a few thoughts on the right way to use technology in the classroom.  The Khan Academy is most known for the comprehensive video library that our founder Sal started creating in his spare time during his hedge fund days. However, thanks to recent funding from the Bill and Melinda Gates Foundation and Google, we have been busily building the software and tools we think teachers and students really need.   

Starting in November, we began a pilot in a few classrooms  in Los Altos, California (fifth grade and seventh grade Math classes).  Bill Gates’ team recently sent a film crew down to document the work and has posted some great videos on his blog, The Gates Notes. The district itself has also created a blog where teachers, administrators, and students are posting their thoughts.  

Sal spoke at TED recently and talked about how we got started and where we are headed.

What’s so different about our approach?  For one, we are leveraging the computer for what it does best and leveraging the teachers for what they do best.  We are ensuring students can truly work at their own pace on their own time. We are making sure students actually master concepts before they move on. We are empowering teachers with the real-time data they so badly need.  We are allowing teachers to make much better use of classroom time, with more peer tutoring, project-based learning, and one-on-one coaching.  Most importantly, we are making learning fun.


Khan Academy Profiles: You are what you know


About a week ago we launched Khan Academy user profiles. Profiles are meant to round out the previous work we’d done on the design of the knowledge map, exercise interface, and badges by bringing all of the information about your performance on the site into a single interface.

There were a few competing design goals here, but the most important one was to flesh out what it means to “progress” within the Khan Academy. At a high level, the page breaks down into a few major sections

  • A set of top level stats that emphasize completion of exercises and videos
  • Vital Statistics: A completely new set of tools for measuring work: Activity, Focus, and exercise completion
  • Achievements: A clearer summary and more interactive badge explorer
  • Recent Activity: Pretty straight forward, but coalesces information that was previously hard to see

The original version of Khan Academy included stars (proficient exercises) and points (a measure of effort). When we added badges, and the badge summary page, it turned into the easiest way to measure progress because that information was so neatly summarized for users. It was never our intent to have badges be a primary motivator for the majority of users, but the lack of a profile was mildly distorting the experience for some users. While badges, stars, and points remain important, we wanted to make profiles a powerful metacognitive tool for students in the same way that the knowledge map helps them understand the interconnectedness of the topics in Math.

Don’t dumb it down

Kids are smarter than we give them credit for. No one likes being patronized at any age, but all too often software patronizes students. Things like overly kid-like color schemes, oversized buttons, and over-simplified user interface are only really appropriate for *very* young children because they send a clear message to everyone about what you think of their level of intelligence/expertise. After spending several hours observing and interviewing 5th graders, I am happy to report that they are more sophisticated and willing to experiment/explore than many adult users I’ve worked with. 

The Focus graph shows how a student has divided their time on the site over the selected time period.

This is a complex graph in some ways, but it also engenders some visceral reactions, like, is it simple looking or busy? A student/teacher might not at first glance notice that on the exercise ring we show a star next to any module name that you’ve earned proficiency in, but when a user hovers the exercise the tooltip will explain what the star means. Our goal here was interesting at a glance but not necessarily completely obvious. The other important design decision on the graphs is that we didn’t use absolute scales anywhere. The result is that whether you spend 5 minutes or two hours on the site each day, the information remains visually interesting and useful.

Make it cool

We want students to start to identify themselves with the information that we’re showing in the profile. We want them to think it’s cool to have finished 80 math modules. As Ben has pointed out, grades are kind of boring in the grand scheme of things. Khan Academy profiles are designed to be visually compelling and highly interactive. Every chart provides tooltips and drill down capability (with back button functionality preserved). This gives a feeling of depth and allows students to explore/discover the connection between different metrics. 

And what’s cooler than success, right? It’s one of the most addictive feelings in the world. So many students struggle with the feeling of failure, and constant absolute measures like grades only exacerbate the problem. Not everyone is going to move at the same pace, and so we owe it to students to find meaningful and interesting progress indicators that include both absolute and relative measures. Profiles were specifically designed to provide measures that help students better understand their own educational progress, set goals for themselves, and measure themselves against those goals. The good news is that it really seems to be working.

What do you think?

In Ben’s original post on what you guys wanted to see in user profiles there were a ton of good discussions and ideas generated. Are you using profiles? If so, are they interesting to you? Missing something important? Are there things you like/dislike about the design itself (especially if you’ve had trouble/confusion while trying to use profiles)?