Discussion about this post

User's avatar
Liam Mcmahon's avatar

Hi Craig, interesting read. I had a couple of questions.

1. You say that “AI tutoring can enhance student learning transfer” but when you look at the 95% credible intervals of the human tutor and supervised AI interventions they are [55.8%, 65.4%] vs. [61.1%, 71.2%], so even if there is a difference between the two interventions, it is not statistically significant.

2. Has this paper been peer-reviewed?

Mike's avatar

I'll admit that I've not read the whole article yet - I've had a skim-read over a few portions of it.

1. Part of the problem of research like this is that there is often a "preferred outcome". Just below the article heading, it shows collaboration between "LearnLM Team, Google & Eedi". Well, at least two of those has a massive interest to show AI working effectively.

2. The questions seem to be multiple choice. How would this work for open-ended questions?

3. Judging by the few AI responses given in the article, it would seem that the questions are fairly easy. How would AI cope with more tricky topics? (Now combine this with point 2.)

4. Example (b) of page 2 shows that the AI hasn't really explained what a ratio is, nor how to form the ratio. It has simply asked the student to count squares, then tells the student the ratio before asking the student to simplify it. The student seems to imply that s/he does not understand where the 8 comes from, judging by the emoji. This would be picked up by a human and explained.

5. Given that the AI responses are them moderated by tutors, the student is essentially getting the benefit of two tutors, not one. Of course the results will show improvement.

6. Teachers/tutors are usually teaching by themselves. This means that if a teacher/tutor makes a mistake, the student is disadvantaged and may not know that an error has been made. If the AI is left to its own devices and not moderated by a human tutor, the negative impact needs to be assessed. This study does not seem to account for that in any way.

7. Of course tutors will also learn things from the AI responses. In mathematics, there are often many ways to explain something, and teachers tend to have their preferred explanation of each topic. If teachers observe lessons from another Maths teacher (especially in another school), they are highly likely to learn something too. I have physically observed lessons in over 12 different schools, across 3 continents (and I've watched many more instructional mathematical videos), and I have learned new ways to explain things.

8. The table on page 5 shows that all of the tutors who were picked for the semi-structured interviews were female. Were all 17 tutors female?

9. There are nuances in communication that a human teacher/tutor will pick up on: tones that indicate whether a student understands as well as gestures, etc.; the way a students' response is worded may reveal a deeper misconception.

10. Ultimately, you cannot get rid of human interaction/teaching. Even if all the issues above (which are by no means exhaustive) are resolved, people need interaction with other humans. Like a child with a new toy, soon reality sets in and the toy loses it's novelty and entertainment value. So too, with AI, students may find it new and helpful at first, but there will soon be a longing for real human interaction, with someone who genuinely cares about them. That human factor will never be replaced, and the longing for human praise and understanding will result in even less attentive students, which will lead to lower educational achievement.

3 more comments...

No posts

Ready for more?