Study Finds Low Engagement with AI Tutor Khanmigo in Tennessee Middle Schools
A two‑year randomized trial showed that most students rarely used Khan Academy’s AI tutor, prompting calls for higher stakes to drive meaningful interaction.
Background of the Khanmigo Trial
Economists Philip Oreopoulos and Nina Low conducted a two‑year randomized study that placed an artificial‑intelligence tutor, Khanmigo, within daily math intervention sessions at 18 middle schools in Tennessee. The AI was designed to coach students rather than simply provide answers.
Key Findings on Usage
96% of the students in the study clicked on Khanmigo at least once.
The median student used the tutor on only about one‑third of the days it was available.
Most interactions with Khanmigo were not focused on math content.
Impact on Math Performance
When compared with peers receiving the usual remedial instruction, the students in the AI‑enhanced sessions showed modest math improvements—similar to gains observed from regular Khan Academy practice without AI assistance.
Industry Reaction
Khan Academy CEO Sal Khan noted in a blog post that the low usage matched the organization’s public statements and emphasized that “motivation matters.” He later told Chalkbeat that the study highlighted the importance of student drive in AI adoption.
Microsoft contributed Azure infrastructure to make Khanmigo’s teacher tools free nationwide, and several states invested heavily: Maryland purchased 4,350 licenses for a two‑year pilot, and Indiana rolled out the tool across 112 schools.
Adjustments by Khan Academy
In response to the findings, Khan Academy announced that the AI tutor would be integrated directly into practice problems rather than remaining optional. The goal, according to Sal Khan, is to make “productive struggle harder to sidestep.”
Why Voluntary Use May Fail
When AI tools are optional, students tend to avoid the tutor that asks probing questions and instead turn to chatbots that provide quick answers. The author of this article reports similar patterns from two years of delivering AI study tools in college and high‑school classes: students consistently deferred engagement when the tool was optional.
Requiring students to use the AI and monitoring their conversations can spark more meaningful interaction, but even mandatory use does not guarantee effort; a student may still provide minimal input.
Assessing the Conversation Itself
Kristen DiCerbo, Khan Academy’s chief learning officer, observed on an EdTech Insiders panel that many interactions were more passive than we would like. She suggests that embedding the tutor may increase the number of conversations, but it cannot create student interest.
The author proposes treating the AI conversation as the assessment. In a pilot study titled “From Tutor to Test,” 21 12th‑grade students at Collège du Léman International School in Switzerland compared strong and weak AI interaction transcripts, built a rubric with their teacher, and were graded on question precision, critical response, and ownership of thinking.
After the four‑week project, 18 of the 21 students reported a change in how they approached AI, and 10 said they now think more about their own input. Confidence in obtaining good AI responses rose from two to eleven students, though some researchers warned that rising confidence could mask over‑reliance on AI.
Personal Experimentation
The author later used AI in two ways for a master’s‑level course: as a conventional Socratic tutor and as a “role‑play” where the AI acted as a difficult interlocutor. The tutor felt boring, while the role‑play was more engaging, especially because the professor evaluated the chat transcripts. Knowing the conversation would be judged raised the stakes and motivated deeper thinking.
Early Classroom Applications
Building on these insights, the author created AI Friction Labs, a platform that uses role‑play “Friction Bots” to test pedagogical value. In a calculus class at Cheshire Academy (Connecticut), 21 students played the teacher while the AI acted as a confused classmate named “Devon.” Students had to explain concepts like “instantaneous rate of change” in plain language. Transcripts were evaluated by an AI engine and the teacher for persistence, clarity, and role adherence. Five students retook the simulation, each improving their scores by an average of 34 points.
Similar positive feedback has been reported by teachers in other schools, though the data remain anecdotal and not yet part of formal research.
Implications for Future AI in Education
The Tennessee study and subsequent pilots suggest that simply providing an AI tutor is insufficient to drive learning gains. Raising the stakes—by making AI interaction a required, assessable component—appears to increase both participation and effort, though further research is needed to determine the optimal balance between motivation, assessment, and student autonomy.
