A 13-year-old 8th grader from Fairview Middle School in Tennessee was arrested and ultimately sentenced to eight weeks of house arrest, a psychological evaluation, and 20 days at an alternative school — all because her school's AI program flagged a conversation she had with her friends on their school-issued devices.
According to the Associated Press, here's the how the conversation went down:
[The girl's] friends had been teasing the teen about her tanned complexion and called her 'Mexican,' even though she's not. When a friend asked what she was planning for Thursday, she wrote: 'on Thursday we kill all the Mexico's.'
Any rational person would understand this wasn't a serious comment.
However, the school uses Gaggle's AI program to monitor chats for threats, and because it's a machine and doesn't understand context, it immediately flagged this sentence as a threat to all the Mexicans.
It's AI, what can you expect?
Rather than doing their due diligence and digging into the context, the school's personnel immediately alerted police, who promptly arrested the bewildered girl, drove her to the police station, strip searched her, forced her to shower on camera, coerced her into signing documents she didn't understand without a lawyer, and threw her into solitary confinement for 24 hours.
Her parents didn't even know where she was, and when they finally found out what happened, they were not allowed any contact until after the solitary confinement was completed.
And a judge later kept up the lack of due diligence and rationality, ruling the poor girl was "probably mentally retarded" and ordering a psych evaluation, followed by putting her under house arrest where she was not allowed contact with anyone but her parents.
And then …
While she was under house arrest, the school filed truancy charges against the parents for not having her in school.
All of this because the girl made a joke that an AI program took out of context. The Associated Press analyzed data from Gaggle and found that its flags were wrong nearly 70% of the time.
Gaggle alerted more than 1,200 incidents to the Lawrence, Kansas, school district in a recent 10-month period. But almost two-thirds of those alerts were deemed by school officials to be nonissues — including over 200 false alarms from student homework, according to an Associated Press analysis of data received via a public records request.
Students in one photography class were called to the principal's office over concerns Gaggle had detected nudity. The photos had been automatically deleted from the students' Google Drives, but students who had backups of the flagged images on their own devices showed it was a false alarm. District officials said they later adjusted the software's settings to reduce false alerts.
Natasha Torkzaban, who graduated in 2024, said she was flagged for editing a friend's college essay because it had the words 'mental health.'
As for why these schools implement these surveillance programs, Anne Costello, a board member of a school district in Lawrence, Kansas, had this to say:
Sometimes you have to look at the trade for the greater good.
P.S. Now check out our latest video 👇
