Hacking computer science education at Khan Academy

Register or Login to like
Register or Login to like
word learn in chalk on blackboard


The following literary transduction is based on a lecture by John Resig given at the Rochester Institute of Technology's (RIT) Center for Media, Arts, Games, Interaction, and Creativity (MAGIC).

I'm deeply honored to be here today. I keep coming back to RIT, every couple of years, and I live in New York City now. There are always new buildings and new things happening. Very fantastic.

I'm here to talk about the stuff I'm doing with Khan Academy. I'm the Dean of Computer Science, which is really a very tongue-in-cheek thing since we can make up our titles. I guess, I'd start with an intro to what Khan Academy is and dig deeper into what I'm doing with CS (computer science) platform in particular.

I joined Khan Academy in 2011 after I worked at Mozilla doing JavaScript tool development. The big thing, our goal, is: bring world-class education to everyone, everywhere, for freeWe are really good at producing educational material. Our math, science, computer science, and art history are very good. Most content is released under a Creative Commons license, so you can take it and do what you want with it.

We're making inroads to bring our content to the world. I've been working on the internationalization of Khan Academy. There are some problems in computer science that are "solved problems," and you'd think doing a website in multiple languages would have been one of these... but it is hard. So many edge-cases... We have videos, articles, exercises, all things relating to our curriculum.

Who here uses Khan Academy's math curriculum as a supplement? (many people raise hands) That's pretty good! We usually tend to skew younger than college age!

Tracking better data

One of the things we do is track all the work you're doing, so you can get a more effective education. Every exercise and video, we track when you did it and how it went. If you got right answers or wrong answers; we can figure out why you're getting it wrong and how to make it better. We have a dashboard where we can push you to work on exercises to move your knowledge forward.

This was the first thing I built at Khan Academy, this framework for easily creating exercises. This is a system where you have a problem, charts, hints, and when you answer stuff, it can steer you when you get things wrong, to give you a better understanding. I worked on the internationalization for this as well, and it was ridiculously hard. Exercises made a lot of assumptions about English and western ways of doing this.

Take "Jane gives one ball to Fred," for example. "Oh, that's easy, we'll just replace the words!" But every language has differing ideas of what is singular and what is plural. Some places have more or less rules and even separate rules if the number ends with a three! We are now available in Spanish and Brazilian Portuguese.

Much of our material on the site is split into a tutorial format. This is where you can go through, watch videos, and do exercises in a very linear form. This is great for our math content, and you see many students at middle-school and grade-school level using this. As I said before, we're tracking lots of data to give students a better experience, but also, the teachers.

Khan Academy is trying to change how we think about education. So we can provide teachers with insights on how students are understanding content and progressing. You can see how they are struggling.

We want to change the "traditional" model where you go in and the teacher gives a lecture to students about a material and assumes that everyone is at the same level. It assumes everyone has equal footing, and often, this is not the case. You have students lagging, and students who are wayyyy ahead. If we had better data, teachers could make more informed decisions.

One-to-one teaching

We break it down for the teacher so they can track specifically what students are working on at a given time, and analyze students in a class or across classes. No longer do they need to give a general lecture. Now it's "Ok, it's math time. You set your goals."

This week a student decides they want to finish long division. They can work on that and watch videos. The teacher can track whether they are getting stuck, and then give a targeted lecture. If there are four students struggling with long division, the teacher can give those four a lecture to reinforce those concepts, rather than doing the same thing over and over again with the entire class. This means they have time to do more one-to-one teaching. They can see who has higher or lower levels of understanding. This is something that provides a huge level of insight to teachers to better use their time.

We have lots of students using this system now: over 3 million problems per day! There are huge spikes in the beginning of school in September—we hit 10 million active students. I'd like to teach on a classroom basis, sure, but this is a way that I can better expand and stretch myself to teach computer science content. Depending how you measure, we have 250,000 to 1 million students doing computer science stuff on the site.

Computer science at Khan Academy 

I started working on the computer science platform in late 2011 and released it in August 2012. We iterated a number of times and settled on this final implementation. The big thing here: this is on the computer science part of the website. We have a lot of curriculum. My coworker, Pamela Fox, is producing all the content (videos exercises, etc.).

We go up to object-oriented techniques, not much overlap with Computer Science 101, and we have a long way to go to be a "full" computer science curriculum. You probably won't be able to go out from here and get a job yet. I want to encourage people, expecially at the target of middle-school and up, in finding the thing that excites them most about programming, and applying it to their life.

Showing people the ways that programming can help improve science, biology, or art. I'd love it if our computer science platform didn't jut produce computer science students, but enabled students to do what they love, with programming to help them do it well. Art historians who can program. Humanities people who can code. Cross-pollination for disciplines that traditionally don't overlap. This is hard, but this is what I'd like to see.

Coding for all disciplines

Who is familiar with GitHub(many hands raise) You can edit code, fork it, upload it, and make changes to the original. Similarly, what we have on Khan Academy, is the ability to make programs. Has anyone heard of Flappy Bird? Someone rewrote this within Khan Academy!

We don't allow outside images yet, so anything you can see here on the computer science site is drawnSo, someone made this game, a student I don't know. It got 700+ votes so far and 1600+ "forks," which we call spin-offs. These are all variations on what has been made. It looks like most of them are about the same. Slight changes in the color of the "flappy bird," stuff like that. Students are taking the code and modifying it. This program has already had over 1000 spin-offs, or programs which were cloned or modified in some way. The top spin-off itself has 113 votes, and over 100 of it's own spin-offs! This is a very different collaborative environment. People making something and learning from it.

Khan's collaborative environment

I wanted to build off of the open source model. I wanted the code to be the front and center and not just show the graphical content, even when it is not a programming exercise.

We have a partnership with NASA, and we're doing lots of simulation stuff. One is making a lander go into orbit and land. It is really hard. Though it isn't required, I wanted to show the code. People who are interested in space can look at the simulation, see it, and say "Hey, how does this work? How can I learn more?" I can go in and modify the simulation, save it as my own spin-off, and make whatever changes I want. That is the model we've embraced here.

Stuff like Minecraft and Cut The Rope style games, those are always popular here. Someone made a drawing program. You can't really save "state," so in the program, at the end, it spits out a giant blob of text, which is what you can copy-and-paste into a program to recreate your drawing!

Students adapt, modify, and change

Students have found ways to work around our system. This type of "meta" programming and activity is what it is all about. Students are even making clubs now. They will hang out and chat with each other in the comments section. Social groups are forming.

Students also tend to make lots of demands and often petition for lots of things. Usually you'll find one at the top of just about every page. One thing often requested is support for playing sound, which I suspect students will abuse plenty. (audience has a good laugh) I like when students can discover things for themselves. What they learn from, it tends to be much more deeply embedded. They take more pride in it.

This has been my baby, real-time injection of JavaScript code, so you can inject "state" into live running code. I feel like it is worth it. You get a much more compelling experience, and you can manipulate things as they happen. You don't have to wait for the program to restart. Even as we we're typing in the code, the program was still running! It works pretty well.

We can record all the actions, and keystrokes, and things being said. We can play back commands and audio. We use videos frequently, but I don't think they are great for computer programming. The thing you want students to do is take some code, pause it, and change things to figure it out for themselves.

We had some really janky solutions where we would pause YouTube videos and regenerate code, but we got to this point. Now, you can pause, make a change, and then when you hit play, it reverts the code, and continues from where you left off.

We also have interactive transcripts now too, so that if you are hard of hearing, or you want the transcript for what has been said, you can get it. And we are now starting to translate those into other languages too (we're working on translating the audio).

All the ability to do this stuff is powered by browser technology, like HTML5. We pull content from Amazon S3, and it works surprisingly well.

I showed you math exercises where a student is given a math problem, they put in an answer, and then they get a "right or wrong" result. That works OK for math, but not really for programming. What works well for learning program is actually writing code. I want the students to be writing out the code. This is one step above recordings. We are basically showing you what code to write, and based on that, they can turn it into their own thing. There are much more complicated programs that can be made from the basics.

One thing I'll mention is that a lot of code is open source and available on GitHub. If you are interested in contributing back, that is all there.

We built a framework this summer called Structured.js. We think it's really cool. You can define a rough structure for the code you want students to write, and then analyze the code to see if it matches a structure. We parse the student code, convert into a syntax tree, and compare it to our syntax tree.

We use jQuery on the front-end, along with backbone.js and a new framework from Facebook called react. It is pretty crazy, and I think you should check it out. I'm slowly getting used to it. On the back-end, we use Python on Google's App Engine. Sometimes we get on 60 Minutes, and the traffic goes bonkers for an hour or two, then it goes back down. Google App Engine handles it quite well.

I think I will wrap up there and answer any questions anyone might have. Questions about what I'm doing at Khan Academy, questions about jQuery of course. (audience laughter)

Q & A session

Q: Does the order in which things are written matter in Structured.js?

A: In this specific case here, it is forcing you to do your 'if' statement before the loop, so yes. A tool like this could be used poorly, sure. And that is why we have such detailed hints. We provide the structure itself as a hint. This is something here, that is one step above a video. They can name their variables whatever they want, we don't care, which is an improvement from most systems like this. This is relatively new, but students love it, and they are using it a lot.

Q: You are doing this teaching in the browser, which is good, but are there any "where to go next?" resources on Khan Academy?

A: We have an article called "what to learn next" where we direct students to tutorials and articles, projects to work on, web development in general, and other language resources. At this point, we are not everything for everyone, but we want to be the launching pad. Here is your first taste, and then you push off from there.

Q: The structure was all S3 and Google App Engine?

A: Yes, all cloud hosted. S3 for file hosting, videos typically pulled from YouTube. We don't have any physical servers. That is the nice thing about being "in the future" now, we don't need any of that. (audience laughter)

Q: What is the history of the name?

A: It was created by Salman Khan. He created many of the videos on the site. We also have other professors and professionals who produce content, but he has done most of it. It started as his YouTube channel to help his cousin learn math. It got bigger and bigger, until it's it got up to like a million people.

Q: How do you moderate all the content? There could be potentially negative things, right?

A: We have a system for people to "flag" content, which puts it into a moderators queue. Once it gets three flags, it gets automatically moderated, and then can be reapproved by a moderator. We haven't had anything too terrible show up yet. It's been running since 2012. Really, it's not the programs, as much as the comments... BIG SURPRISE! (audience laughter) There have been "factions" of middle-school students who battle in the comments sections. We were worried in the beginning, but this hasn't actually happened yet. We banned outside images and only allow images that we provide. We had a student write a program that turns an image into a multi-lined call to the rectangle function to recreate it. If we find a way to ban something, students will always find a way around it. They are mostly doing things like putting up pictures of their favorite Pokémon.

Q: Biggest tech challenge at Khan Academy?

A: The real-time stuff was pretty huge. The internationalization stuff of the past year was really hard, but hard in a different way. Not hard in the "how do we scale" way, but in the "understanding the cross-cultural issues" way. One of the hard problems for his platform, was making it in such a way that young students can understand what is going on. We ran a whole summer school and did lots of playtesting. We'd get a different batch every week. They'd get summer school credit, and we'd get bug reports. We were in there with our notepads, and running everything through JSHint. We were trying to provide errors that were much more intuitive, but this wasn't explicitly a tech problem.

Q: You talked about "clubs" forming in the comments, I was wondering how far you were willing to take the pseudo-social aspect? Are there plans to expend into different languages or other types of programming?

A: Yes, in some areas. One immediately I think of is humanities. I didn't think about this initially, but the way in which you are making these programs and building a creative work, you spend lots of time and you want people to collaborate. This can happen for writing, poetry, art, music, all of these things for which there isn't a model for teaching online. Having the staffs and bars for composing music and hearing the playback in real-time would be great. Even better would be for live-forking and editing for others to collaborate. More immediate feedback on what projects students build is a goal.

The only other language we're looking to go into is HTML/CSS/JavaScript. The huge advantage of JavaScript is that it can run natively, you don't need to pass it back and forth. For now, we'll be sticking with browser native languages.

Q: Khan Academy is free, right? How are you making money right now, and do you see any change in the future?

A: We are funded by grants. The Bill and Melinda Gates Foundation, The Carlos Slim Foundation, and others. There is a lot of interest, and funding, and we aren't in the position of every other education startup—where you're charging teachers, schools, or students money. I don't want to be pinching pennies out of students pockets. For additional revenue, we have contracts for folks who are using our content in commercial contexts.

We do have lots of jobs, tons of internships. The whole computer science platform was built by me and interns, so if you wanna work with me and build cool stuff, please do.


This article is based on work at http://ejohn.org.


At the Fedora Project Remy served as Community Action and Impact Lead, bringing more heat and light to the distro's user and contributor base.

1 Comment

Why mention Flappy Bird (which is breaking the rules) and the spin-offs (which people think they are original programs because they don't look at the Based On) when instead you can mention another game like <a href="http://www.khanacademy.org/cs/falling-pixel/1053970737">this</a> or <a href="http://www.khanacademy.org/cs/a/1106141208">this</a>