AI in Education #5: Five Things I Learned from Simon Woodhead about What a Decade of Student Data Can Actually Do
This is the fifth in a series of AI in Education specials, where I speak to the world’s leading experts about how AI is changing teaching and learning.
Simon Woodhead and I have been working together for the best part of 15 years. In that time, we have co-founded two companies, collected tens of millions of student responses to multiple-choice diagnostic questions, and built some spectacularly fancy data visualisations that nobody used.
Simon is the Chief Data Scientist at Eedi. He studied Bayesian statistics for his PhD about 20 years ago (he assures me it is feeling rusty, but I have my doubts). He is the person who, when I came to him with my idea for a website where teachers could share multiple-choice questions, thought it was a terrible idea but helped me build it anyway.
That website became Diagnostic Questions. Diagnostic Questions became Eedi. And the data we collected along the way – which for years we could not figure out how to monetise – turns out to be one of the most valuable things in education AI right now.
It was long overdue to have Simon on the podcast, and I wanted to share five takeaways from our conversation that I think every classroom teacher and school leader will find interesting.
1. The data your students have been generating for years is suddenly the most valuable thing in the room
Simon and I started collecting student response data back when we launched Diagnostic Questions around 2012. Every time a student chose A, B, C, or D on one of our multiple-choice questions, we captured it. Every time they wrote an explanation for their answer, we captured that too. We also captured whether they said they guessed.
For years, we had no idea what to do with this data. We could summarise it – here are the most common misconceptions on expanding brackets, that kind of thing – and I used it for Guess the Misconception talks and blog posts. But we could not build the kind of predictive models that would make the data properly useful at scale.
What changed? Two things. First, machine learning methods matured to the point where they could handle the volume and complexity of our data. Simon described how, back in 2020, researcher Chung Zhang at Microsoft had built a model for healthcare – predicting the next best action for a doctor treating a patient – and Simon Peyton-Jones (one of the smartest people Simon has ever met, and someone who has been instrumental to Eedi) saw the connection to education: what is the next best question a student should answer based on their response to a previous question? That collaboration kicked off our first serious machine learning work.
Second, the design of our questions turned out to matter enormously. Because our distractors are designed to capture specific misconceptions, a student choosing option B is not just “wrong” – it tells you something specific about what they were thinking. That additional layer of information is what makes the data useful for building models that can predict not just whether a student will get a future question right, but which specific wrong answer they are likely to choose.
If you are a teacher who has been using diagnostic questions with your classes – whether on Eedi, Diagnostic Questions, or even just on mini-whiteboards with well-designed distractors – you have been generating this kind of data for years. The difference now is that we have the tools to do something with it.
2. A model can now predict how your students will answer questions they have never seen
Eedi now runs what Simon calls a dynamic quiz. A student answers 10 multiple-choice questions. Based on their responses, the system predicts how they would answer hundreds of other questions they have never seen – including which specific distractor they would choose. And it does this, according to Simon, with a level of accuracy that genuinely holds up when tested against real student data.
How? The system is not just looking at right and wrong. It is looking at patterns across distractors, response times, and the relationships between questions in Eedi’s knowledge graph – a taxonomy that maps maths all the way down to what we call constructs. A construct is something like “adding fractions where one denominator is a multiple of the other” as opposed to just “adding fractions.” Each construct is linked to specific misconceptions, and each misconception is linked to specific distractors. So when a student chooses a particular wrong answer, the model learns something precise about what they know and do not know.
I pushed back on this. My concern is that it becomes self-fulfilling. The model predicts that a student will struggle with adding and subtracting fractions. The teacher sees the prediction and intervenes. The student then does fine on adding and subtracting fractions. So was the prediction wrong? Simon was honest: this is genuinely hard. As he put it, “as soon as you have intervened, you have affected the data.” The team is building more complex models that can account for the difference between a student’s first attempt, a retry, and a post-lesson checkout – but it is an ongoing challenge.
What matters for teachers is this: the model is not trying to replace your professional judgement. It is trying to give you better information before you walk into the room. And a quiz that can do in 10 questions what used to take a full diagnostic assessment is, if it works, a Godsend.
3. AI can finally do the things we tried to do ten years ago and failed
One of my favourite parts of our conversation was remembering all the ideas Simon and I had that were ahead of their time – and by “ahead of their time”, I mean they crashed and burned.
Take student explanations. On Diagnostic Questions, we built a system that let students write explanations for their answers, and other students could read and like the best ones. The idea was beautiful: crowdsource understanding, let kids learn from each other’s reasoning, surface the best explanations to the top.
Can you guess what happened?
Yes, kids drew penises using dashes and dots on the keyboard. Other kids liked them. And suddenly, the most popular explanation for quadratic equations was a picture that definitely was not about quadratic equations. We built swear filters. Kids treated it as a challenge. We shut the whole thing down.
And yet the idea was sound. A student being able to explain their reasoning is such an important skill, and an incredibly useful resource. We just could not solve the moderation problem at scale.
Simon’s point is that we can now. Natural language processing models are good enough to flag inappropriate content before it gets shared. The same is true of OCR – optical character recognition – which we tried years ago to extract text from the thousands of questions that teachers had uploaded as PowerPoint screenshots. Back then, it could pull out individual words but could not link sentences together, and it was hopeless at reading mathematical notation. Now it is brilliant. Simon’s team has gone back and extracted the text from thousands of legacy image-based questions, making them accessible in a way that was impossible when they were first uploaded.
The lesson here goes beyond Eedi. If you had a good idea for using technology in your classroom five or ten years ago and it did not work, it might be worth revisiting. The underlying technology has changed fast enough that what was genuinely impossible may now be straightforward.
4. The best AI tools for education are built with a human in the loop – and the human is a teacher
Simon described a model for AI-assisted question writing that I think strikes the right balance.
Here is how it works. A teacher (or a content designer – at Eedi we call them learning designers) asks the AI to generate a multiple-choice question on a given topic. The AI does a reasonable job. It gets the question stem right. It usually gets two or three of the distractors spot on – distractors that genuinely capture common misconceptions. But it almost always gets at least one distractor wrong. Maybe it produces a wrong answer that no real student would ever choose, or it misses the most important misconception entirely.
So the teacher edits it. They swap out the weak distractor. They tweak the wording. And here is the part that Simon loves (because he is a data scientist): the system captures all of those corrections. It records what the AI suggested, what the teacher changed, and why. And that data feeds back into the model, making it better over time.
I have been experimenting with something similar myself – using Claude as a thought partner for question writing, where I write a quiz and then ask the AI to evaluate it as if it were a psychometrician. But Simon’s version is more structured because the corrections are captured systematically, not just in a chat window.
This connects to a theme that has run through the whole series. In my conversation with Daisy Christodoulou, she argued for 10% human judgments and 90% AI in assessment. Adam Boxer worried that if teachers stop writing their own questions, they will stop developing as professionals. Simon’s model threads the needle: the AI does the heavy lifting, but the teacher’s expertise is still essential – and is captured as training data to make the system smarter.
5. Taking your data global is far harder than translating the questions
I saved my favourite example for last because I think it illustrates something important about AI in education that most people have not considered.
One of the most popular primary maths questions on Eedi shows a girl counting in words: seventeen, eighteen, nineteen… what comes next? The correct answer is twenty. The distractors are twelve (because it starts with “twe”, just like twenty – a phonetic trap) and ten-teen (because students think the teen numbers continue in this way). It is a cracking question. The data shows a genuine split across all three options, which means each distractor is doing its job.
When we sent this question to our translators for a project in Rajasthan, they came back and said: We cannot use this. None of the distractors work in Hindi. Hindi does not have “teen” numbers – each number in the teens has a unique word. And twelve and twenty do not sound similar in Hindi, so the phonetic trap disappears.
So you cannot just translate the question. You have to redesign it from scratch for the new language – which means you need to understand the specific misconceptions that Hindi-speaking students have about counting, which may be completely different from the misconceptions that English-speaking students have.
Simon calls this domain shift. A model trained on millions of UK students does not automatically work for students in Rajasthan or Texas. The knowledge graph – the taxonomy of constructs, misconceptions, and prerequisites – has to be rebuilt or at least adapted for each new context. Some nodes overlap across countries. Some do not exist at all. And some of the prerequisite relationships flip, because different countries teach topics in different orders.
This matters for anyone thinking about AI in education at scale. The promise of AI is personalisation – the right question for the right student at the right time. But “right” depends on language, culture, curriculum, and the specific misconceptions that students in a particular context actually hold. There are no shortcuts here, and anyone selling you a one-size-fits-all AI tutoring product for a global market is probably oversimplifying the problem.
Over to you
This was a brilliant conversation, and I have only scratched the surface here. You can listen to or watch the full episode on the Mr Barton Maths Podcast. And if you are interested in the data science side of education, check out the Data Science and Education meetup that Simon runs every month or two.
This is the fifth in a series of conversations I am having with leading thinkers about AI in education. Here are the others in the series:
I realise this episode is a bit different from the others in the series – less of a debate, more of a behind-the-scenes look at how the sausage gets made. But I think the details matter. If we are going to talk about AI in education, we should probably understand what the data looks like, where it comes from, and why it is so hard to get right.
What do you think? Is this the kind of thing you want to hear more about, or do you prefer the big philosophical battles? And has your school started thinking about the data it collects on students and what it could be used for?
Thanks so much for reading
Craig
🏃🏻♂️Before you go, have you… 🏃🏻♂️
… checked out my brand-new book series: The Tips for Teachers guides to…
And checked out my all-new, ad-free, Mr Barton Maths website
