What do 4 million students' responses tell us about mathematical understanding?

Diving into the data on constructs and misconceptions

Have you checked out our brand-new small step quizzes that align to the White Rose Maths schemes of work for Years 2 to 9?… No?… then here you go!

For many years, my Diagnostic Questions and Eedi websites have collected data on student misconceptions. We are currently working on making this data—and, more importantly, the actionable insights that stem from it—available to you. Watch this space—I promise you will be the first to hear about it!

In the meantime, I want to share some of my favourite things that the data has revealed.

How we collect and tag data

As you may know, we initially assess student understanding using multiple-choice diagnostic questions like this one:

We then tag each question with a construct, which are the micro-skills that are the building blocks of mathematical understanding. The question above is tagged with the construct: Subtract two-digit numbers mentally, where renaming is required.

There are a lot of constructs in maths. Here is a snapshot of all the constructs under the heading of Place Value:

We then tag each distractor—the carefully chosen incorrect answers—with the misconception it reveals. So, in the question above:

• Answer A may be selected by a student who believes you always subtract the smallest number from the largest number.

• Answer C may be selected by a student who has mistaken the subtraction symbol for addition.

• Answer D may be selected by a student who has failed to rename the tens digits.

We selected 12,000 of our best diagnostic questions, tagged each with a construct, and mapped the 36,000 distractors to the associated misconceptions. We then collected over 4 million student responses and analysed the data.

Below are some of my favourite findings.

Misconceptions by Year group

Mixes up squaring and multiplying by 2 is an example of a misconception that remains prevalent across high school, but falls with age. It seems to be one of those things that requires regular exposure over time.

Whereas Mixes up the x and y values on the coordinates actually increases in prevalence as students get older! Is this because we don’t explicitly teach it past a certain age and treat it as assumed knowledge in more complex concepts older students encounter?

Misconceptions by income status and gender

At Eedi, we are especially interested in narrowing the achievement gap between students from low-income backgrounds and their more affluent peers. Therefore, we wanted to see if there were any examples of misconceptions that were more prevalent in students who are registered for free school meals:

Each dot represents a misconception - I have highlighted one for you.

The answer is: no. The misconceptions are clustered tightly around the dashed line, indicating an equal prevalence for both groups of students.

We see something similar when we compare the misconceptions of girls and boys:

This is good news. It means we don’t need to differentiate our instruction on any of these grounds, and instead can concentrate our efforts on delivering a high-quality learning experience for all our students.

Misconceptions across concepts

Misconceptions are not unique to constructs. We often observe the same misconception appear across several constructs.

Let’s take one of the misconceptions from the diagnostic question at the top of this post: Subtracts the smallest number from the largest number. We have already observed this misconception appear in the question assessing students’ ability to subtract two-digit numbers mentally. But we also see it in a question assessing students’ ability to subtract integers with different numbers of digits using a formal algorithm (answer C):

And when subtracting decimals with a different number of decimal places (Answer B):

And when performing division calculations with standard form (Answer D):

If this misconception is not identified and addressed early in a student’s mathematical journey, it will hinder the learning of the many concepts upon which it is built.

Misconception cascades

We are particularly interested in how misconceptions relate to each other. Does a misconception in one area lead to misconceptions in other areas? Knowing this can help us identify the root causes of issues, which also has implications for curriculum ordering.

We identify such relationships using a machine learning model, which we then test with teachers via surveys. This allows us to produce misconception cascade diagrams:

• Each node represents a misconception.

• If they are connected with an arrow, it suggests causality between the two misconceptions.

• If they are connected with a dashed line, it suggests a correlation.

• If they are connected by a faint red line, they are unrelated.

In the plot above, we observe the following cascade:

Does not understand the term multiple > Confuses factors and multiples > Does not know the definition of a prime number > Believes 1 is a prime number

I’ve loads more I could share with you, but I will pause there as I am definitely straying into geek territory. But I want to know what you think:

Was there anything you found particularly interesting?

What would you like to know more about?

What questions do you have?

Let me know in the comments below!

🏃🏻‍♂️ Before you go, have you…🏃🏻‍♂️

1. … checked out our curated collection of the very best diagnostic question quizzes?

2. … signed up for my free online workshop: 25 Tips for Teachers?

3. … read my latest Tips for Teachers newsletter about cognitive endurance?

4. … listened to my most recent podcast about Atomisation?

5. … read my write-ups of everything I have learned from watching 1000s of lessons?

Thanks so much for reading and have a great week!

Craig