School AI surveillance can lead to false alarms, arrests
apnews.com
apnews.com
In that sense, it isn’t “normal”, it’s just “something that’s happening in theory but eh maybe it only affects scary people or whatever idk”. I feel like this tolerance we’re developing for outside forces invading “private” spaces, nominally for these loose justifications of harm reduction, will be what _actually does_ make it normal.
Once it’s truly normal, and people think it’s what keeps them safe from mass shootings or whatever, it will be too late to get rid of it. I think fear and normalcy will motivate its spread to places beyond school chat platforms and Snapchat.
Yeah, she'd probably get fired but law enforcement wouldn't get called because there was no specific or credible threat and the company doesn't have the mandatory reporting requirements that the school does.
> ...
> Gaggle’s CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said.
> “I wish that was treated as a teachable moment, not a law enforcement moment,” said Patterson.
Of course they didn't do things as the Gaggle CEO describes. They were legally compelled not to.
I love a good complaint about corporate meddling as much as anyone, but sometimes it really is the government to blame.
The purpose of a system is what it does.
https://www.astralcodexten.com/p/come-on-obviously-the-purpo...
More discussion:
https://www.astralcodexten.com/p/highlights-from-the-comment...
Here is a counterpoint:
https://rivalvoices.substack.com/p/the-purpose-of-posiwid-is...
Pretty obtuse on law enforcement, who don’t honestly do much regarding crime that doesn’t incarcerate people, and does not close the circuit on his similes, but entrenches behind the strawman projection “the system must have been designed by evil people who were deliberately trying to hurt you, and so you should become really paranoid and hate everyone involved.”
I usually like that author’s train of thought but this is insufficient reasoning.
If you take this view, then a fair assessment recognizes that "the system" in this case consists of more than just the Gaggle software.
Look at public schools as just another municipal enforcement department tasked with making sure the kids all meet state standards and it makes sense (or at least to the same extent that all the other indefensible shit done in the name of government makes sense).
Anyway they took us to look at the inmates in the halfway house who were behind bars and then they could come out to the bars and as their part of the whole scared straight exercise they would of course yell stuff at us, which was mainly about how they wanted to have sex with the approximately 16 year old girls on the trip.
Gosh, Utah sure is a morally upstanding place.
> on Thursday we kill all the Mexico’s
Especially considering the frequency of violence in American schools, can't really blame the school for jumping to conclusions.
Of course you can blame the school. They were too lazy to look at context and determine if the threat was real and credible. They took the determination of a complex tool as an unquestionable fact. The system supplies the fact that the user's account made the comment. All other facts need to be made by investigation. This statement provides a reasonable suspicion to investigate, but should not exhibit probable cause for an arrest as it requires a threat be credible, incite panick, etc per the specific terrorist threat law. This requires investigation and thought.
Like if you're gonna do "make the news" shit you oughta at least have someone check your goddamn work first.
Do you think whoever is doing this at the school is a qualified professional, e.g. a child/teen psychiatrist that knows the kid in question well enough, to be able to determine if the threat was real and credible?
If you want to solve this problem solve it at the root, not by overreacting to teenagers.
It's terroristic threats. That's the law most of the school shooting threats would get charged under. The real problem is that most states have automatic reporting laws, which means you have to report anything that sounds like a threat even if it isn't. This is the main difference between regular cases and school cases - you end up with a lot of junk being reported and potentially causing more harm than it was intended to prevent.
There is an easy way to stop 99.9% of all school shootings, and it isn't 'automatic reporting laws'.
https://www.nea.org/nea-today/all-news-articles/arming-teach...
https://www.ue.org/risk-management/premises-safety/increased...
Don't worry, the people who want these Orwellian things will find a more "marketable" way of describing it.
> When a friend asked what she was planning for Thursday, she wrote: “on Thursday we kill all the Mexico’s.”
> Taken to jail, the teen was interrogated and strip-searched, and her parents weren’t allowed to talk to her until the next day,
> She didn’t know why her parents weren’t there.
> A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl.
Wow, 2025 is wild. Police and court should have psychological evaluation instead and maybe some time off without pay to cool off. Protecting children, no matter how many of them they have to traumatize and incarcerate.
Not the sharpest bullets in the barrel...
Back when I was in school the web filtering got so draconian those of us in the know started keeping Tor browser on a thumb drive, and carrying it to whatever computer we were using. Some went as far as to reboot from a Linux image on said thumb drive. That was nearly 20 years ago, can't imagine it's any better now
1. Schools running and monitoring their own communication platforms. This seems fine.
2. The US government monitoring private communication platforms like Snapchat and arresting kids on school grounds. This seems bad.
The way it’s written though mixes these together and makes it look like schools are monitoring the private communication of students. I think it’s fine for schools to monitor there own platforms but weird for the government to monitor all platforms haphazardly.
I'm not saying this doesn't happen, but I am saying that this has never happened to me or anyone I know.
How long before they use AI to do that?
AI hype is at play here as well, not only in the breathless press releases from AI companies.
2. The US government monitoring private communication platforms like Snapchat and arresting kids on school grounds. This seems bad."
Public schools are government.
A student said, “on Thursday we kill all the Mexico’s,” on a schools private communication platform. The school should correct that behavior.
Unfortunately they involved law enforcement. Thats where I see the problem. A better solution would be detention and informing the parents.
Sounds like they could've reasoned that they face the least chance for liability if they pushed the responsibility to law enforcement.
When you were a kid gun laws were stricter.
Some states have fewer laws, others have more. At the federal level, there are more laws and rules than there were in the 60s or 70s (overall more than at any prior time).
And of course enforcement varies. I remember many people coming to school with guns in their cars during hunting season even though it's not legal.
One can impute it from the content assuming it’s relevant, i.e. in America.
As for the feature bans those are clearly municipalities attempts to get around either constitutional or political impediments to the bans people actually want. Magazines fed semi automatic rifles are what people want to ban, banning rifles with muzzle shrouds is how they get there.
Studies on this topic are fraught because the gun industry has long prevented the normal research funding issues on this topic and have fought tooth and nail any data collection efforts.
Yes there's more legal access federally than in the 90s but the difference is pretty much wholly on a state by state basis with some states having no or slight change and some states having large change.
> If we are only looking at the federal level, then there has been no substantial reduction.
There _was_ a significant reduction at the federal level. They acknowledged that and then changed the goalpost.
I then claimed
> Studies on this topic are fraught because the gun industry has long prevented the normal research funding issues on this topic and have fought tooth and nail any data collection efforts.
This is true. The Dickey Amendment prevented first the CDC and then the NIH from collecting gun violence statistics from 1996 until today. Though in 2018 they were able to add a rider to it to make it a little easier.
FOPA makes it impossible to collect registration information for federal use, including in data exchange for studies around gun ownership.
I'm not sure what more you want me to backup. Would you like the actual legal citations on those?
Screeching about whatever fed law changes or lack thereof the Brady Campaign told you to screech about is pointless. Fed law directly affects only a tiny minority of buyers because people buy what's available and no recent federal law changes have increased/decreased the legal buyer pool and sales volume.
You don't need the CDC or the NIH or whatever other "authoritative" source who's boot you prefer the flavor of in order to make assessments of how things have changed over time. State law changes and sales data are very easy to come by. Those are where the real meat of the change is. Many states over the past 20yr went from it being a hassle to "just toss a pink Ruger in your purse" to a simple retail transaction.
This forced reporting necessarily creates false reports. Under the law, things like terrorist threats are required to be credible or incite panic. Reporting things that aren't credible is arguably a violation of law under any other context, yet they choose to ignore that with these mandatory reporting laws. Basically it creates a situation where nobody is allowed to use their brain - automated conveyer to the criminal system.
Brain users get jaded and a) stop b) seek alternate employment.
Schools don't just crush kids.
Eh, terminal violence was potentially threatened. Calling the cops seems fine if no teacher or administrator can vouch for the kid. (Particularly if, as is true in this case, the law requires “any threat of mass violence against a school to be reported immediately to law enforcement.”)
To the extent someone fucked up, it’s the cops who allegedly caused the 13-year old to be “interrogated, strip-searched and spent the night in a jail cell.”
> Police are a volatile, uncontrollable solution
No they don't and shouldn't be.
> If you introduce police into a situation
I think informing the police about a dangerous attitude in some child isn't the wrong solution. The police not showing up at the parent's place and sending a therapist, but instead treating a child like a criminal adult is criminally neglect.
Also right here in the comments we have somebody mentioning that people behave how you treat them. That doesn't apply soly to children. If you treat the police like a militia and don't show respect they will behave like that to you.
> I think informing the police about a dangerous attitude in some child isn't the wrong solution
I was regularly beaten at home by religious zealots, before finally being kicked out onto the street and becoming homeless at 16. The police were over at my house every three weeks or less, constantly threatening me with foster care and juvie if I didn't respect my grandfather's right to beat me.
The man would routinely make me choose a weapon, often but not always a belt. He would then beat me savagely with the metal buckle like a whip, all over my body, until I would stop crying, because "men do not cry, I'll teach you how to be a man", and "I'll give you something to cry about".
Every ounce of state intervention made it worse, because of police and administrative corruption. Don't even get me started on how they surveilled me through my school.
Yeah, we do have ideals. And regulations, which... don't always match those ideals. And then we have reality, which... doesn't always match regulations. To ignore reality and say that the regulations are the true reality is honestly a very ignorant and dangerous thing to do, especially when you try to instill this perspective into others.
You are one citizen. The police represent the other million citizens. They have the monopoly on violence, so that all the other citizens can't do that. I think this is way better than every citizen caring weapons and doing self-justice. Even when the police is going crazy, that is still better than civil war. They do have checks and regulation even if they don't always work, the other citizens haven't.
That being said, I can absolutely see, how you distrust the police. I can't imagine what justification the police gave to their superiors in your case. I would expect public outrage if this became public, but maybe your country truly doesn't care.
Of course you shouldn't report such issues to a corrupt militia, which your police seams to be posing as. However that is not what I would call a police and it should be reported to the police if you had such a thing in your region.
When you ignore children with violent intentions this will lead to a stabbing or shooting later on. The police will treat a single claim very differently than recurring threats for violence. Informing the authorities early one makes them able to send therapists the first time. When they only ever hear about problems once things went violent or even only at the time of the amok, they need to send the violent force on the first occurrence, because they can now only try to shield the victims not help the delinquent child.
Police exist to enforce the laws within specific parameters. They are basically a modern formalized take on midlevel men at arms. They mostly only de-escalate conflicts and prevent harm incidentally.
A concept of a written law is very different from a rule by decrees and the will of the ruler. The difference is that deciding whether something is lawful is outside of the ruler or the enforcer. (i.e. separation of powers) This is what differentiates a liberal democracy from a dictatorship, but also 19th century and modern monarchies from absolutism. So no police is not modern men-at-arms.
Somebody should tell them that.
> If they don't you should file a disciplinary complaint.
... which will be ignored. They may or may not laugh in your face.
I wish I lived in your dream world.
I was personally given a maximum sentence of 6 months in jail as an underage, first-time offender because a police officer who had been stalking my friend group planted weed on me at the scene of an accident. I was denied the right to a trial by jury, they refused to let me out of jail until I'd signed a paper giving up that right. I was homeless and needed to graduate high school, so I had no choice. My defense attorney was then forced off my case so that I could be given another attorney under the thumb of the prosecutor and judge's racket, and so this was never addressed.
Seven cops came up one after another and gave wildly different testimonies of what happened, then the judge gave me my sentence because "I think you're lying". When I moved to appeal, my compromised attorney refused to let me, saying, and I quote, "The judge is my boss. If I let you appeal his decision, he'll make my life miserable."
See, the prosecution was pissed off that I tried to fight my charges, and so they worked with the judge to give me maximum possible jail time, despite being a homeless kid who had never been arrested before.
And the officer who stalked me, planted weed on me and arrested me is a known methamphetamine producer and distributor. I have personally witnessed her roll up in her cop car with a toothless old woman in the passenger seat, and watched that old woman grab as many boxes of pseudoephedrine as she legally could and bring them back to the cop car. This is called smurfing.
Presumably it was on a day when her regular chemical supplier was unavailable, and presumably they hit several such stores that day in order to amass a large supply. Some people I know literally murdered her brother over a meth dispute, and the police found a giant meth lab on his property.
Oh, and the mayor was the prosecutor's dad. So, can't tell the DA, can't tell the mayor... Maybe the state police? No, they work with them. FBI? No, I have left anonymous tips several times and nothing ever came of it. The only investigation I ever caught wind of mysteriously dissipated.
I even went to journalists, but no one took it seriously and those who did left it alone.
Everyone in that town knows the racket. Everyone. The mayor retired a few years ago, the prosecutor is good friends with my extremely abusive grandfather, and no one will ever answer for my mistreatment.
The article specifies this is a Tennessee law. The school is in Kansas.
Agreed that the cops screwed up, but the school is also responsible.
Welcome to global privacy trends in 2025.
But also, is the school creating a blindspot for itself where it has _less_ information than if these systems were not in place? From one of the court documents linked in the article:
> The District and Gaggle—acting at the District’s behest—do not merely infringe students’ Fourth and First Amendment rights—they do so in a way that actively undermines the very safety concerns the District claims to address. Acting with the District’s knowledge, Gaggle automatically and unilaterally seizes student materials containing an undisclosed list of “trigger words” or phrases, without regard to context. This sweeping censorship extends to all student documents and content within the Google Workspace/Suite on District platforms. As a result, when students use their school-issued accounts to seek help or report concerns—particularly about mental health—Gaggle frequently intercepts and seizes those communications before they can reach their intended recipients, including teachers, counselors, and even parents. Such censorship denies some students the help they may need, contrary to Defendants’ articulated purpose for using Gaggle.
... i.e. the school is inadvertently quashing students attempts to ask for help or report problems
> When The Budget’s editor-in-chief investigated via requests under the Kansas Open Records Act why Gaggle continued to operate in violation of the purported exemption for student journalism, the findings responsive to her requests were, ironically, suppressed—apparently unbeknownst to the District officials tasked with responding to the requests—by Gaggle. Like the fox guarding the henhouse, Gaggle prevents parties from learning about how and when it operates.The District was made aware of Gaggle’s actions with respect to student journalists, but took no action to eliminate the problem.
... i.e. the school is generally unaware when it is itself silenced by its "safety management" software, even when this brings it out of compliance with legal requirements.
https://www.documentcloud.org/documents/26040279-lawrence-bo...
I think most of us can agree that if kids are using Chromebooks or something provided by the school district they should be held accountable (similar to how if I say something awful in a work chat or email I'll be held accountable.
It is the overreach like you said that is the problem. People thinking that we "must protect the children" at all costs is the problem.
The specific district here has a "no bullying" agenda which, as a father formerly with kids in the district, is not a thing. But they promote it and they are the top performing schools in the state so whatever they can do to pander to parents is what they will do.
There's also a large Protestant population here with mega maga churches, and moms against liberty (real name: Moms for Liberty, but I like to call a spade a spade) is heavily enmeshed in the school system. Whatever the parents say goes, and this is just another example.
I always told my kids: I don't care who is asking - a teacher or the police. The answer to "show us your phone" is always "you need to call my parents". I am so glad my kids are no longer in the school system here.
Many of my friends are teachers, lots in this district, and I have to say the state of education has really gone backwards in general and specifically here
>>> A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl.
If one assumes that the court did take into consideration context and age, it appears to largely validate the follow up decision once flagged. (I don't agree with lack of parental contact, to be clear.)
If the police were willing to make an example out of someone, the judge they work with is likely to do that as well.
Oh, really? Do they have data that shows a significant reduction in violence since surveillance started, or is this just reframing false positives (that can result in arrest, eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school) as a net benefit. My money is on the latter.
The victims will still be victims, they’ll just be punished by the system even harder for being a victim.
Immediately and automatically engaging law enforcement, and even the FBI, is horrific. Kids have always had greatly restricted freedoms in schools, but transcending the classroom and monitoring their digital lives is just training them to accept the surveillance state.
Sure, these people exist. They are dangerous.
Same thing with unfettered capitalism, the systems only work if we continue to support said systems. When the rules break down, so do the desires of the collective to maintain said systems.
If we can somehow win back trust in our collective ability to democratically solve problems... that should solve the problem.
I think that involves some creative solutions to collective decision making.
> teaching enough people that economics aren't the only metric for running a society, community, and life.
I agree that this is the major problem with society today, but I don't see a solution when this is exactly the desired state of affairs by anyone with any amount of money.
It's funny how only in economics is a secondary effect (efficiency/production/profit) optimized for, and we just hope or assume that it will result in wide-scale health/happiness/wellbeing. In any other situation, we would just design the system around the desired outcome.
From the perspective of those pushing this kind of technology and political movement, is that a bug or a feature?
We already see it with the modern surveillance state, post 9-11 the US citizenry has lost so many freedoms and if you ask random people on the street about it they would be perplexed. Hell, even my friends give me a bit of the "ahh so this is your conspiracy theory" look when I mention them. Growing up through 9-11 and the forever war was pretty dystopian, or at least a March into the dystopia's that I only read about in books.
I think the problem is that people send kids to public schools and just hope for the best. Imagine you have a brand new child, and you send it to school, and the child ends up saying something offensive, is this the child's fault? I think not. The child was trained on harmful data, it's not surprised the child exhibited undesirable behaviours.
So, the principal, one Roman Peredun calls me up and says that my son used a bad word. I asked him what word. He wouldn't say it. So I asked how am I supposed to know how 'bad' my son is if you can't even repeat the word. He then spelled the word. I said, oh, 'fuck'. Yes, that's not in dutch however so he must have picked it up in your school. Peredun hung up and I sent my kid back to school the next day.
Ya know what? No sane person gives a fuck.
Your comment reduces children to entities that will behave as expected provided they get fed “good” data.
Humans are not LLMs.
There are plenty of studies on formative environments, especially on how negative environments can lead to negative behaviors.
Here we have a bad joke. The system flags it. The school sends it to the police. The police detain and interrogate the kid. Everyone is treating the determination of a complex automated system as their own determination. We also have every actor treating this as a credible threat. For this to be credible, you have to have the means to accomplish it. They gave a timeline. You know you have time to investigate before making an arrest. Problem is, nobody cares.
To retaliate, the next day administrators had metal detector wands waiting for us right off the busses, took every single cellphone they found and locked them up at the school board office for the rest of the year.
That school was absolute hell, a battleground between students and teachers. I am not exaggerating at all when I say that being spotted outside your classroom was an immediate expulsion, with not even enough time between classes to pee or use your locker (two minutes, we had to run). As part of the escalation, the fire alarm began being pulled at least twice a day. Any student who had even a moment alone with one would pull it immediately. Absolute chaos and a direct result of power collapse due to a racist, authoritarian school board that only knew how to wield institutional violence.
For this reason, my kid will always have a phone in order to protect themselves from administrative abuse. I will fight for that tooth and nail. I had over 40 write-ups in just elementary school for refusing entertain abuse from authoritarian staff, and I'll be a failure if my kid doesn't walk the same path.
Each time you'd walk into class, it was a dice roll if another piece of equipment had been stolen. We stopped buying and using projectors because any new ones got stolen within days. Sometimes I'd walk into class and every single piece of furniture had been completely turned upside down. Students would play on their phones, yell, throw things at the teachers, and the teachers would just ignore them.
It was also an extremely racially charged situation which had a big impact. I think there were less than ten white people in the school, the rest of the school was black, and a handful of hispanic folk.
The white people were almost all racist, and so apart from my neighbor, I only hung around black people. As the only white person there who hung out with non-white people, my nickname across the entire high school somehow became Tarzan, in part due to my long hair. When I'd walk down the halls each day, people would beat their chests and make ape noises and give me dap.
I'll never forget being at an award ceremony and walking up for a few awards, each time the entire student body would start stomping in the bleachers and making jungle noises and shouting my nickname. The visibly perplexed administration had no idea what was going on.
Because my credits would not cleanly transfer to their curriculum, I had two free periods each day and used this time to help out the administration and teaching staff, running errands, organizing, doing paperwork, etc. This allowed me to interface often with the school board, who was... entirely comprised of white, racist, old men. Men who fundamentally refused to understand the predicament facing the student body, and who were too eager to escalate instead of finding a way to bridge communication between the student body and faculty. Instead of empowering students, they saw them all as future bodies for the prison and service industry, and it was obvious. Students learned nothing, but would get straight A's. They would of course absolutely fail state and national standardized tests. They were shoved through the meat grinder with nothing to show for it.
The kids used a school communication program to say something racist. Schools should monitor school communication platforms.
The only thing I disagree with is the level of punishment (sending a kid to jail for a night).
However, there should be a way to address the risk and prevent acts of terrorism without turning into a police state. A more reasonable response might have been to have the parents come by for a long meeting with the principal, school counselor, and school resource officer to talk about the severity of making such statements - even if she did make it sarcastically as tone/intent is difficult to judge. Then suspend the student for a day or so and have the school resource officer periodically check to ensure she didn't bring any weapons.
To play devil's advocate though, I can't imagine the stress from parents who have children of Mexican descent in such a situation.