The Ethical Minefield of AI in Classrooms

The First Time a Teacher Let an AI Grade the Essays

It happened quietly. A professor at a large university uploaded 47 student papers to a GPT powered tool. The AI returned scores, comments, and even suggested rewrites. The professor saved six hours. The students never knew.

This is not a hypothetical. According to a systematic scoping review of 118 peer reviewed papers published since 2017, researchers Lixiang Yan, Lele Sha, Linxuan Zhao, and Yuheng Li found that large language models are already being used for 53 distinct educational tasks, from grading essays to generating quiz questions to detecting plagiarism (Yan et al., 2023). The review, published in the British Journal of Educational Technology, is the most comprehensive attempt yet to map where AI is actually being deployed in classrooms and what happens when it is.

What the authors found should make every parent, teacher, and administrator stop and think. Not because AI is failing. Because it is working. And nobody has figured out the rules.

How Do You Know the AI Isn't Just Making Things Up?

The Yan team reviewed papers published between 2017 and 2023, covering everything from automated feedback systems to AI tutors that predict which students are at risk of dropping out. They categorized the use cases into nine types: profiling and labeling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation.

The most common use case? Content generation. AI writing lesson plans, creating practice problems, even drafting entire lectures. The second most common? Grading and feedback.

Here is the problem. The authors found that most of these systems suffer from what they call "low technological readiness" (Yan et al., 2023). That is a polite way of saying the AI is not reliable enough to trust unsupervised. A grading system might give a B+ to a well argued essay and an A to one that is grammatically perfect but intellectually empty. A content generator might produce a perfectly coherent lesson on the French Revolution that quietly invents a key date.

The review documents cases where AI generated feedback was "inconsistent or inaccurate" in up to 30 percent of outputs. Not a rounding error. A systemic flaw.

The authors also identified a deeper issue: replicability. If you run the same student essay through the same AI system twice, you might get two different grades. The models are probabilistic. They do not remember what they said last time. For a teacher trying to be fair, that is a nightmare.

Who Owns the Student Data?

This is where the review gets genuinely uncomfortable. Yan and colleagues found that "insufficient privacy and beneficence considerations" were among the most common ethical failures across the papers they reviewed (Yan et al., 2023).

Think about what happens when a student interacts with an AI tutor. The system records every question, every wrong answer, every hesitation. That data is gold for model training. It is also deeply personal. A student who struggles with reading comprehension, who reveals anxiety about math, who types slowly because of a learning disability: the AI sees all of it.

The review found that most studies did not specify how student data was stored, whether it was anonymized, or whether students had given informed consent. In several cases, the data was sent to third party API services with no clear privacy protections.

This is not a theoretical risk. In 2022, a major edtech company was caught using student essays to train its language models without disclosure. The company settled with the FTC. The practice continues.

The Yan team recommends that future systems "adopt a human centered approach" and "embrace the initiative of open sourcing models and systems" (Yan et al., 2023). In plain language: schools should know exactly what the AI is doing with student data, and they should be able to inspect the code. That is not happening now.

The Grading Problem No One Wants to Talk About

Here is a specific finding that should haunt every college administrator. The review found that AI grading systems are reasonably good at scoring essays for surface level features: grammar, structure, keyword density. They are terrible at evaluating argument quality, creativity, or original thinking.

Why does that matter? Because students figure it out fast. If the AI grades on structure, students will optimize for structure. If the AI rewards certain keywords, students will stuff their essays with those keywords. The AI does not just measure learning. It shapes what learning becomes.

The authors call this "gaming the system" and note that it is not a bug. It is a feature of any automated assessment tool. The model cannot tell if a student actually understands the material or just learned to write essays that fool the grader.

This is not a new problem. Multiple choice tests have the same flaw. But AI grading feels objective. It outputs a number. Teachers trust it. Students adapt to it. And the thing that actually matters, the messy, nonlinear process of learning to think, becomes invisible.

What the Research Does Not Prove

The Yan review is a scoping review, not a meta analysis. That means it maps the landscape rather than calculating effect sizes. The authors do not claim that AI is always harmful or always helpful. They do not prove that AI grading is worse than human grading. They do not show that students learn less with AI tutors.

What they show is that the evidence base is thin. Most of the 118 papers were small scale studies, often using custom built models on narrow datasets. Few tested the systems in real classrooms over a full semester. Fewer still measured long term outcomes like retention, critical thinking, or student motivation.

The review also does not address the most obvious question: what happens when the AI is better than the teacher? That is not a joke. In some controlled experiments, AI tutors have outperformed human instructors on specific tasks like drilling vocabulary or explaining math procedures. If the AI can do that, should we use it? The review does not say. It only warns that we are not ready for the answer.

The Three Recommendations That Matter

The Yan team ends with three concrete recommendations for anyone building or buying AI for classrooms.

First, update existing systems with state of the art models. Many of the tools in the review were built on older versions of GPT, which are significantly less capable than GPT 4 or its successors. Using better models reduces errors and improves reliability.

Second, open source everything. If a school cannot inspect the code, they cannot verify the model's behavior. Proprietary AI in education is a black box that makes ethical oversight impossible.

Third, adopt a human centered design process. That means involving teachers, students, and parents in the development of AI tools. Not as test subjects. As co designers.

These recommendations sound obvious. They are not being followed.

The Hidden Assumption Nobody Checks

Underneath all of this is a deeper problem that the review hints at but does not fully articulate. The entire premise of AI in education rests on a specific model of learning: that it can be measured, quantified, and optimized. That more feedback is better. That faster grading is better. That personalized pacing is better.

What if those assumptions are wrong?

There is a growing body of research suggesting that the most effective learning environments are not efficient. They are messy. They involve struggle, confusion, and the slow construction of understanding through repeated failure. An AI that optimizes for correct answers might actually undermine that process.

The Yan review does not resolve this tension. It simply documents that the field is moving forward without asking the question.

What This Actually Means

• ▸Do not automate grading without human oversight. The AI can handle grammar and structure. A human must evaluate argument quality, creativity, and original thinking. Anything less is educational malpractice.

• ▸Require transparency in data handling. Schools should demand written policies on what student data is collected, where it is stored, and whether it is used for model training. No policy means no deployment.

• ▸Test AI tools the way you test new curriculum. Run a pilot. Measure outcomes. Compare with a control group. Do not assume that faster feedback equals better learning.

• ▸Train teachers to be AI literate. They need to know what the model can and cannot do. They need to spot when the AI is wrong. They need to explain to students why the AI gave that grade.

• ▸Build systems that can be inspected. Open source or nothing. If the code is secret, the system is not trustworthy. This is not negotiable.

The Yan review is not a warning to stop using AI in classrooms. It is a warning to stop pretending that the ethical questions have been answered. They have not been asked.

The professor who saved six hours by letting the AI grade those essays did not think about any of this. He just wanted to get home earlier. That is exactly why the minefield is so dangerous. Nobody sees the wires until they step on one.