Going Dark, and the era of law enforcement hacking
blog.cryptographyengineering.com
blog.cryptographyengineering.com
That resulted in the Communications Assistance to Law Enforcement Act, which mandated that central offices offer remote wiretapping. Capacity up to 1% of lines is required.
Back in the electromechanical era, the only call data that could be collected was outgoing dial pulses, using a "pen register".[1] (The one shown in Wikipedia is mine. It's a beautiful piece of antique brass telegraph technology. It records dial pulses as dashes, and has to be wound up like a clock, with a big brass key.) The Supreme Court decision allowing "pen registers" without a warrant refers to these "extremely limited" devices. That definition has been stretched and stretched by law enforcement into all non-voice data collected by telcos.
Law enforcement still wants more.
In the fictional world of The Wire, one of the "wires" were blown when the guy checked on his mobile bill to find out his account was flagged to not be disconnected. I'm sure this was art imitating life or inspired by type of thing
Many criminals even today don't engage in basic counter measures like you see on that show.
And I'd call them "dysfunctional" more than inept. Bad leadership stymied by political goals overcomes even good policing.
Doesn’t help that a big chunk of the competent criminals have fully or partially internationalized their operations which acts as a very strong shield against LE.
The politicians wanted the immediate appearance of effectiveness over real effectiveness, top police brass wanted to keep their jobs, and rank and file police got to do ineffective work that made little difference.
It's a nice morality tale, and I'm sure there are instances where it is absolutely true. But that it is true as a rule, or even that they are comparable levels of analysis (one are "white collar" crimes and the parallel would be "blue collar" crimes, no? But street crimes and immigration offenses aren't working mans crimes - these are "no collar" crimes: anyone can commit an immigration offense, even rich people. And street crimes are often drug driven.
This analysis by Gottschalk and Wacquant suggests a "conspiracy" tone within a class war context. I would want to know if they didn't have some class war bias in their analysis before I accept their analysis from authority without examination.
It’s easily the best LE show that exists, but not just because of its accuracy; it’s incredible in its pure honesty. Some of the cops are total worthless shit bags, worse than the best of the criminals easily. Allowing that overlap, one that exists in real life, injects a whole lot of nuance into the story that is basically nonexistent outside of the show. And allows characters to grow and improve. It takes writers with actual LE experience (which they had) and a lot of talent to write that kind of thing without totally blowing it.
The billing accounts never would have crossed the same desk,
let alone been crossed.
The lines also weren’t terminated at the suspects’ homes. So, again, no vector for the bill to be sent there.
I’d lean towards “myth” on that - until a more convincing story appears.
They claimed that since it never got shut off when they didn't pay it must be tapped.
Possibly correct I suppose...
The big boogeyman that noids have insisted on for the past ~30 years is that intel agencies are saving everything they can to try and decrypt it later. Thankfully for us, the rate at which internet traffic grows year over year massively outpaces the global production capacity of spinning rust.
The last time I did this napkin math I think the collective global production was barely enough for some small fraction (maybe 1/8th or 1/12th) of everything.
It isn't exactly something that can be done in secret. Everyone else has to buy hard drives too, there are only so many sources of raw materials, places that can refine them, etc
E: I think the most relevant thing to avoid if this is a concern for you would be commercial VPNs. There's a reason fiveeyes doesn't want them to be banned. They can almost assuredly save piles of money by simply coercing providers into allowing them physical access to leverage side channel attacks
neat!
https://upload.wikimedia.org/wikipedia/commons/e/e3/Pen_Regi...
assuming you still have it, it wouldn't be the worst idea to update the pic to something without chromatic aberration and motion blur. the wiki pic is probably the worst of the bunch compared to others on google image search :)
I bought it on eBay years ago, out of curiosity, and fixed it. I built a little control box to go with it, with a phone dial and a 9V battery, and found a source of paper tape and ink. So it can be demoed. If it ever is used in a court appearance or legislative testimony, that would be better than it sitting in a box.
I was nerd-sniped by the above link (which I've added spaces to, to keep it from being truncated) and thought "Huh, since when does Wikimedia Commons use tracker barf?" So I found the page for this image:
https://commons.wikimedia.org/wiki/File:Pen_Register.jpg
then copied the full-size image's link... and sure enough. I guess they want to keep tabs on sites that hotlink their images and don't even practice good link hygiene.
Any idea of museums that may hold historical pen registers (or paraphernalia) for public viewing?
The mechanism is that of a "Morse inker", a recorder used to receive telegraph signals. Here's one that's part of a Siemens telegraph set from 1862 or so.[1] The wiretapping version is the receive part of a telegraph set made portable in a nice wooden box with a carrying handle.
All this shows how rare wiretapping once was. Law enforcement had to use repurposed antique telegraph equipment.
[1] https://collection.sciencemuseumgroup.org.uk/objects/co32910...
This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite.
My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.
Over the long term this state of play could change, and IC/LEO organizations could start leaning more on application vulnerabilities than on platform RCEs. But the action would probably still coalesce around a couple of app-layer targets that could themselves be hardened.
According to a friend, who is working for an telecom company in Europe, Lawful-intercept is done at a server rack provided by and managed by a law enforcement agency, all traffic of this telecom company is copied to this server rack. The server rack also has a second direct connection to the law enforcement agency. Employees of the telecom company don't know which phone numbers are targeted or which traffic is intercepted.
To list a few: - WebKit (CVE-2016-4657) - WhatsApp (CVE-2019-3568) - Chrome (CVE-2021-38003)
Then, also the distinction between platform vs application gets fuzzy with bugs in PassKit etc.
This does seem to be true in the vibe coding era, which I expect will implode at some point. But LLMs could certainly lead to a future where vulnerabilities are scarce. The best time to vet security factors is designing the model and writing the initial code, and LLMs are extraordinarily useful for this too. Most code with security implications is not written by a security expert.
An LLM can be the most anal and well informed security expert you can find. If you write the code yourself but have one in the loop from the start, the code will be in much better shape.
That said if companies want to "ship ship ship fast", then yes even AI can produce bugs or regressions if not carefully reviewed by the human.
These days we're generating multiple times more code than we were writing before. That means a similar multiple of opportunities for bugs to be introduced - so the ability to automate security review is more impactful in proportion to that.
You should question your ability to see any bugs on YOUR side.
How do you know? You might be adding (latent) bugs every time your LLM fixes one for you.
That covers functionality - it doesn't catch the kind of bugs talked about in TFA.
If the first derivative is positive, adding people does not help.
https://en.wikipedia.org/wiki/1/2_%2B_1/4_%2B_1/8_%2B_1/16_%...
Speak for your workplace only.
Most places still consider a PR or commit yours, if your name is attached.
Act accordingly.
I've been looking for a new job where people aren't so reckless... the AI psychosis is heavy at my current job.
Author claims something slightly different but that’s how I would look at it.
The extremes are more potent because either you get super secure 10 times vetted system or you get underpowered model slop with all the vulnerabilities it entails.
I think this is a side-effect of the old product management process adapting to AI. We (as an industry) were never very good at defining features rigorously, because there was a smart human in the loop who had to implement the feature and could push back on sloppy definitions.
Whereas security bugs are easy for the LLM to define and fix.
Any engineer who's skill value evaluates below 0 on my scale, ends up with a large negative number when they use AI. Anyone with a positive number ends up with a large positive number.
The extra bugs you're seeing are from devs on the wrong side of 0 on my poorly defined scale.
It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.
Anyways, while there I attended a little roundtable on software security. It was me, representing a small unit from the Air Force, some dude from Google, and like 15 game devs.
Despite only being a dev of 5 years at that point, I was SHOCKED at the lack of knowledge on software security. Even simple concepts seemed completely foreign to the game devs, though the Google dude seemed to have a really solid understanding of security.
Obviously game devs and website devs and all kinds of devs have different focuses, but it just blew my mind that out of all the topics there I might’ve been considered a comparative expert in, security was somehow the one. I wasn’t sure if that was a major plus for military devs or a major concern for the other devs, but now I’m starting to learn in the latter direction.
As an end user, security in game development seems to hit a minimum is the early 2000s.
I recall playing EverQuest and using ShowEQ on a Linux machine on the same network as a proxy to silently eavesdrop on the traffic to show a map of the game with all the hidden/obscured/visible stuff all present. They either never encrypted that traffic or I quit playing before they did.
As multiplayer games started to become the default, netcode improved quite a bit and basic encryption was implemented - but I don't recall a single multi-player game, ever, that was free of hackers when playing online.
It feels like over the years the issue moved from remote exploitation to hiding processes on the user's machine. That has lead to kernel-level anticheat systems. I'm unsure of the effectiveness as when they started to become common I moved all of my gaming to dedicated hosts because I wasn't willing to give that level of access to my "real" computers. That said, I've certainly not noticed a decrease in the perceived number of cheaters.
These days, as far as I know - and to be clear, this isn't an area of deep interest for me - I'm not worried about my gaming accounts being compromised through technical means. It feels like that sort of security is a solved issue overall, although I suppose it's also possible that other forms of exploits have simply gotten easier/cheaper.
...ish. The solution is false positives and a permanent underclass. You can't play those games unless you're on windows and you grant root to the anti-cheat.
I don't care much about that, but i do care about people being unemployable because some amorphous machine doesn't like the way they comb their hair.
Last century, if you grew up five to a room then at least you knew why you were marginalised. It wasn't just, but it was legible. Now, some of us are living in Terry Gilliam's Brazil.
BTW, I highly encourage all EU citizens to prepare Cyber Resilience Act complaints for September 11, 2026 when the law kicks in ;-)
[1] https://www.reddit.com/r/blackops2/comments/1v3xowq/just_ent...
Google has probably the best security team in the world, and they need to. The military deals with state actors by definition.
Gamedevs need to ship. I'm not saying they don't care about security, but short of some showstopper critical problem, it's not a thing and doesn't need to be.
It doesn't matter if your game is secure if nobody plays it lol
Besides stopping cheaters, in almost all conceivable cases, security doesn't make a game better.
I could see a future where the government is replaced by the software system which has the fewest security flaws, simply because it is the last thing left standing as everybody hacks everybody else. We end up with a Matrix-like or Terminator-like future where humanity now lives in service of the machines.
The cops will find a way, the criminals will find a way. It's ultimately people v. people with access to the same level of technology.
we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software.
if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks.
and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny.
freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes.
I think it's a mistake to think they can't - rather, they won't.
Still?
Sorry.
Especially curious what is the term for the current type of Government in the USA?
It's entirely different than any prior one historically speaking, isn't it?
I am just beside myself at such an idea that people looking to feed the prison machine cannot as easily find excuses to turn normal citizens into prison feed.
Just super sad guys.
What felonies are those normal citizens being convicted of that's landed them in prison?
I get the basic 'not wanting everyone's comms to be monitored' part. I think there's an important conversation to be had because indeed, that can't be the answer.
But it sounds like you're saying most people who are convicted of felonies based on a jury unanimously being convinced by the electronic evidence against them... those people are unjustly in prison?
Certain people are arrested and detained far often than others, because of who they are and not what they did.
Many positions in the justice system prioritize conviction rate above all else.
There are many problematic relationships between legislators, law enforcement and the for-profit prison systems.
Our position in this list should make anyone afraid:
https://en.wikipedia.org/wiki/List_of_countries_by_incarcera...
If LE decides you should be "taken off the streets", regardless of if you did the crime you are accused of, you are likely going to prison for something unless you are wealthy. If you are wealthy, they may not bother. All very interesting.
There is plenty of distrust in LE and the legal system on the right. There is also plenty of people who believe it all is OK because "those people" are ending up in prison. You will never reach the people who do not care until it happens to someone they know, or someone like them.
Like, yes, the system works at the level you're assessing it. The problem is that it's based on a mountain of flawed premises and so we've ultimately created a literal torment nexus with a smiley face taped on top of it and everyone gets to wash their hands of it and go home: justice served.
The danger of constant surveillance is that a lot more of these "being a dick" incidents will be prosecuted when politically convenient (because equal, fair prosecution of all incidents would not happen as that would quickly cause the law to be changed, preventing its future use selective enforcement).
It shouldn't be so, but it is. It's human nature; the powerful use that power to defend their interests, unless prevented from doing so. The people without power or favor get watered down justice, if at all. The system isn't architected around this, but it's not architected to prevent it either.
Everyone is legally entitled to this protection.
And you consider this to be definitionally true; or do you have specific evidence of Greg Lairmore doing something that merits comparison to that regime?
Or do you actually believe that "people who entered a country without authorization and remained there long term with the intent of defying all local immigration law, should be extradited, by force if necessary" is comparable to killing a significant fraction of an entire ethnic demographic, worldwide, while simultaneously waging expansionist land wars?
Why would the "best and brightest" work for governments that pay a fraction of market price?
> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.
> While I think this is great, for law enforcement and offensive intelligence agencies, it’s going to be a nightmare.
> So what do we do about it? I honestly have no idea. [...] it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today. [We’re] just going to have to hope that this time we make the right choices.
In a nutshell, what I’m saying is “They shouldn’t do either” is at best a platitude and at worst a distraction preventing critical thought.
In a nutshell, this is bullshit. Bootlicker bullshit.
In a democracy (which our Western nations are certainly supposed to be, though that's become less true over time), if the people in government are doing things that are counter to the interests of the people, the appropriate response is to replace those people. Protest, contact your representatives and senators, contribute to campaigns that oppose what they're doing, etc.
And, of course, in an authoritarian state, if they decide they're going to add in backdoors to our phones, or ban encryption, or take whatever other draconian measures, then it doesn't matter a huge amount how we, the people, express our opposition to this; it's extremely unlikely that we have just enough collective strength and will to prevent them doing A, but not to prevent them from doing B instead. Either we can stop both, or neither.
Governments should behave. I know they don't, but they should, and the population should do everything in their power to force them to. What other choice do we have?
Please demonstrate your powerful faculty of critical thought and explain what we should do instead.
Like SMART goals, I believe the other commenter was pointing out that platitudes lack specific, measurable, achievable, realistic, and timely aspects required for organizing and achieving progress.
So is a backdoor worse than a security vulnerability? I don't know but that is what this article is about.
https://gitlab.com/here_forawhile/tmail/-/raw/main/tmail3.pn...
https://i.postimg.cc/0yKXWSbg/IMG-20260815-113546.jpg
Yeah follow along I still have alot of testing and polishing to complete before I call it v1.
No license on yet either. Still pondering that as well.
Also, be sure you are sending a plaintext copy (formatting stripped) using multipart/alternative, as most terminal mail clients can’t do HTML mail or can’t do it well.
One of the author’s blind spots here is the concept of “digital sovereignty”. The US is continuing to ban more and more Chinese-made hardware out of fear that the Chinese govt has installed backdoors in them… which you could interpret as an admission that the US does exactly that with American made products. Globalized supply chains are, erm, complex, and few if any companies are really going to be able to achieve “digital sovereignty” with hardware. But with software it’s actually plausible, though obviously hard. Whether or not the feds have actually installed backdoors in Microsoft Outlook, foreign govts are rightly concerned that they have, and are increasingly pushing to avoid US made software for simple national security reasons.
> I don't have to outrun the bear, I just have to outrun you.
It's like when the bear can't catch the slow guy anymore so he starts setting up traps and finding ways to cut off legs. Now the faster guy actually has to deal with the bear.
I wonder what will happen with YC and this news-forum as it's owned by Altman too.
I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.
Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?
I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.
That's precisely the author's point: deliberate backdoors will be more adversary-exploitable than ever before, but the demand for such from law enforcement agencies is likely to ratchet upwards.
But that's one step after, I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases? Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it? How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
If you're a US company building an app/device/etc. such that an intelligence agency like the CIA or FBI would want access to the data in that product which is normally secured, then they're not going to try to sneak it in there without your development team knowing. They're going to have a meeting with the owners of the company and say, "hey, we'd really like you to implement this backdoor for us, and in return we won't cause you in problems."
Note that this does certainly already happen a lot, but it's also not something that can happen across the board (like the author points out). Apple, being one of the largest companies in the world and who has one of their biggest selling points being their security, has explicitly refused to do this to the point that the intelligence agencies couldn't break into an iPhone until another company found a way to do so.
>I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases?
The people working on these codebases are "in on it." Of course, we're talking backdoors which are very subtle, target very minimal infrastructure, and are known about by very few people. But, like I said, companies are currently shipping products with backdoors in them knowing they exist already. AI doesn't change that dynamic.
>Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it?
The only people who would have access to the systems that can even be flagged by this stuff would be people who would know about it. Keep in mind that the scale we're talking here is massive. Think about how software development works at companies like Apple, Microsoft, Google, etc. There are devs working in offices all around the world where they only ever have access to a fraction of the code that company owns. These companies are very capable of keeping their stuff locked down. It's a necessary component of their work.
>How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
Hopefully my explanation at this point is clear, but just to be concrete: backdoors are, by design, very hard to detect. That doesn't mean they're just sneakily written code that humans don't notice as they read over it, but, instead, they're very subtle implementations in very specific parts of huge systems that are already locked down to the point that the number of people who even have access to those portions of the systems are very limited. These agencies don't slip in backdoors without anybody noticing; they convince the minimal number of people needed to know about it to implement it. Again, we're talking about a meeting between the directory of a three letter agency and a CEO, where the CEO then directs the CTO to implement the backdoor who then instructs the handful of very high-ranking engineers to do so.
AI systems in these companies may very well flag these backdoors to the people who already know they exist, then these people can tell the AI "hey, those are their on purpose, so just move on," and the other 99.99% of the company will never know they exist.
I suppose it's important to emphasize, again, that these systems are incredibly massive and complex and most people at these orgs don't have any access to most of these systems, so it's not like you can expect an intern running BugBot across a repo and expecting it to find a backdoor.
I would imagine most big companies, like Microsoft, have dozens of CIA and FBI plants in their organizations. Agents who are legitimate software engineers, tasked with acquiring intelligence and undermining security.
And then you say, loudly and publicly, "all the source code of our software is public, and our binaries use binary transparency so it's not possible for us to build a binary that doesn't match the source, and people will rapidly find this in our source code at which point we go out of business and you stop having a product to backdoor in the first place".
(And you move out of the US.)
And since this is a foreseeable future, you should start acting now to prepare for that future.
In fact, we're increasingly seeing a desire to build the "backdoor" directly into the software, e.g. mandatory age verification, client-side scanning, etc.
> (And you move out of the US.)
And then everybody claps.
Name me a software or hardware company—one big enough that the US government would actually care to force them to add a backdoor—that would be willing to give up the US market? The only one that's shown the least bit of backbone is Apple, and while I like them and appreciate what they've done in that vein so far, they're never going to move to open source software running their stuff, and they're pretty well embedded in the US, and very, very unlikely to try to move regardless of the headwinds there.
I'm fully with you that this would be a wise and moral thing to do, but frankly, our tech companies are neither wise nor moral. They are self-serving, greedy, and many of them have wanted to become the neofeudal overlords of a new order since before Trump started smashing the old one.
Certificate Transparency has essentially eliminated the problem of backdoored CAs, because attempting to do so would destroy an entire CA. Binary Transparency can do the same for software.
Companies have to follow the laws of the countries they operate in, not just the countries their physical headquarters are in.
That's why, for instance, Apple has to follow the DMA in Europe.
Furthermore, especially for many of the tech companies, where they are located is an integral part of their culture. They are Silicon Valley. You're going to have a very, very hard time convincing any of them to up stakes and move.
And further-furthermore, move where? Europe has, unfortunately, made similar authoritarian noises (eg, Chat Control). China is already more of an authoritarian state than even Trump's USA. Ditto for Russia, and, AIUI, India, though both in somewhat different ways.
Apple is subject to the DMA in europe because 1) they have employees in Europe and 2) they want to import and sell physical phones into Europe. It's easy to block imports or fine employees. It is not especially easy to prevent people from spending money on a service provided entirely via the Internet from another country.
It is not at all impossible to impose sanctions on an entire company. But if the US tried to do so over a refusal to put in a backdoor, and the company very loudly made that clear, that would to some extent be exceptional free marketing.
It's seems like an odd-duck for sure, and I doubt it's a realistic proposition. I do think "perfect encryption for dummies" is all well and good until organized crime organizations are able to challenge the government in certain regions of our country.
All of this sends us deep into the realm of political philosophy, the nature and purpose of governments, and the freedom vs security tradeoffs we live with.
I listened to the latest Plain English podcast this morning, which was explicitly about the potential for a ransomwarepocalypse in the coming years, as open models let any tom, dick, or harry become capable of a plug-and-play ransomware attack, instead of that being left to the realm of professionals. It's a bit nerveracking to think about every nigerian prince scam suddenly becoming a sophisticated attack on your local water sanitation system.
I think I can kind of see that from the text re-reading it, it's kind of hidden though and not so explicit if this is what he is trying to communicate. Thanks for giving me a new perspective to read it with!
The author agrees with you and addressed this point. The US forcing backdoors into its own systems ("own": Those for sale and distribution within the US) would create vulnerabilities making those systems weaker.
Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base.
But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.
Now it’s just as you said- asking AI to red team your app will get you pretty far.
> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.
His conclusion sounds extremely optimistic to me.
Perhaps we are going to go up a level with hacking done by probing the systems and the system of systems.
Pre-AI, the advantage went to the entities with the largest budget to hire the best and brightest security engineers.
Post-AI, it'll go to the entities with the largest inference budget.
Right now we're in a transitionary period where it's kind of a tossup which approach is more practical, but at the end of the day - it's still all about how much money you can throw at the problem. I'm just hoping the threshold climbs high enough it's no longer practical for governments to be able to compromise individual actors' devices because doing so would waste a 0-day that's far, far more valuable than prosecuting one arbitrary person is worth.
There’s (almost) always someone else out there that is willing to do it, and there’s (almost) always a dollar amount that you can’t turn down.
(It’s of course not only the US, just that the largest AI providers are US based and we know from history how US agencies operate)
I'm using open weights models on a privacy-focused provider right now, and that's adequate for my current usage, but I'm rapidly getting to the point where my agent's access level to my data (and to a lesser extent, my accounts) is becoming something I'm not comfortable sending outside my network at all.
My hope is that models that are roughly on par with Deepseek V4 Flash can be run on hardware that I can own for <~$5k in the near future. We're close, but not there yet as far as I know.
The only long-term solution to this is self-hosting.
An actual objection I had while talking to an aerospace company was they don't want opensource models because the threat of it having a poisoned training example on specific systems.
It's easily the most hidden malware possible, completely undetectable until an exact set of tokens unlocks it. Is it line 100,543 of your security product? You will literally not know until it plants it in there.
If you can't stop them from sometimes telling customers things like "yes, I will give you a penthouse suite at our hotel for only $3/night", why would you be able to guarantee that, with some specific set of tokens, they would produce a perfect and undetectable backdoor customized to the code at hand?
Anthropic has done such checks at the agent level: https://cybersecuritynews.com/anthropic-claude-hidden-code/
Their excuse was defending against distillation attacks but you can see how that can be abused
Open models also let you read reasoning traces. That means anomalies would show up when the backdoor activates, like a run of unrelated words or a jump in top token probability. It's only undetectable until the first time it happens.
Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it.
Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.
Crime's been decreasing for years, and (from what I understand) this year is tracking to be one of the safest years on record?
Definitionally, there absolutely has to be a correlation (not causation) between those two factors you listed.
Most of the crime decrease from its peak happened before the year 2000.
That means back in the days of law enforcement needing to do low-tech wiretapping techniques, crime was still rapidly decreasing.
There’s just not even a correlation. And, as a reminder, correlation is not causation even if it was there.
Strong emphasis on the legal system part, but you've completely dismissed that private communications frequently are required to address causes of issues and the word subpoena makes it rather clear that refusing to comply comes with penalty.
These laws exist - they aren't the focus yet. Right now there's still no need; just hack the device or compel the cloud service to give the data, why waste energy getting consent from its owner!
More bugfinding AI, more end to end encryption, more CVEs and more fixes, cannot happen soon enough.
150 years ago was the invention of the telephone, and I think that articles like this seem to assume that prior to this, police just never caught any criminals.
A year when police were relatively rare, and quite new in the western legal experience, having only emerged from slave patrols in the United States and from the founding of the 1829 Scotland Yard in the United Kingdom. This is a year with living memory of a time when the state did not employ people to do what was ostensibly the civic responsibility of every person to quell crime and protect others.
heh. long greasy slide. It really does feel like that.
Apart from the obvious harms of invasion of privacy, and fishing expeditions being biased to the places you decided to fish. There is the simple fact that data can be misleading, especially without context. An interceped communication is a piece of data that is intrinsically tied to the trust of the inteceptor. A few people with an agenda can collaborate to create a seeming truth by 'discovering' the same thing from different sources.
Requiring warrants compelling information holders to provide data, not only serves the task of protection from abuse but also create a record of provenance that can be verified.
It also provides a degree of symmetry in capabilities which discourages actions that one party may do over another if they are motivated to act because they have a temporary advantage over another.
I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.
Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones.
You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.
If they can get access to all our shit, at least make them say it out loud and put in the effort to get legislation passed. There's no upside to the current situation.
And it's not like they're not trying to introduce backdoors already... Seen the whole age verification thing?
It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.
Basically in the AI age, the difference between a vulnerability and a backdoor diminishes..
We will certainly see “front door” access as various places have or are attempting to enact client side scanning, id verified online access, software root of trust remote attestation, and the general attack on e2e.
Taken together your identity tied to unmodifiable software, necessary to access the web or modern economy with client side scanning and no e2e … is a front door.
All this has nothing to do with ai, bugs, hackers or security vulnerabilities, let alone backdoors
Exhausting infrastructure vulnerabilities even without quantum could be a game changer for many technologies and enable things we can’t do right now, like vote on our phones.
Either way there’s other realms it’s relevant like armed drones policing a street. Without a way for the public to activate a shutdown sequence, it becomes very difficult to stop if it starts murdering innocent children on some glitch. Which you cannot have because then the whole purpose is ineffective. Likewise digital communication is purposeless and face competing mediums for sure
There is NO WAY the US government will let most of the software world go dark.
In the same manner, if the critical concern is a device/service being used against one's self remotely, stop the use of the device/service if it is capable of remote access or data collection for later retrieval.
The ultimate baseline reality is that deciding to care whether or not remote access/data retrieval is using a method supported by the device manufacturer/service provider and/or whether doing so is allowed by the current legal system is too late.
Premptive HN disclaimers: I acknowledge the growing unavoidability of devices and services supporting remote access and/or data collection as I type this here on my phone. Remote access and data collection are very convenient and somehow still expected by device owners and service users to be under their control alone. It can be difficult to determine if a device or service supports remote access/data collection. There are often immediate consequences and eventual legal consequences for stopping the use of devices or services that belong to others. Working to change the legal system to slow/reduce use of devices and services against one's self and encouraging others to do the same is admirable.
It will be interesting to see if my prophecy becomes reality.
BTW I also hate that Hacker News is being dominated by articles on A.I. lately. Maybe we should vote on HN reducing or even eliminating A.I. related news?
If automated pentesting in PR review CI pipeline will become table stakes - which is very plausible - maybe the OP has a point.
I just hope that politicians don't jump in and say "we know how to fix it", because unfortunately where they put their hands, they just make things worse (regardless of which country)
Developing modern 0-day zero click RCE exploits chains like those developed for iOS is far more complex than hacking to generic company servers
It's completely different scale of difficulty factor. Not a single documentated case of of any AI tool developing such exploit exists
There is a reason why those mobile 0-day exploits are sold and bought on exploit gray market for as much as 10-20 million dollars each
I'm quite in favor of impenetrability of software as the alternative is constant friction, to be honest.
No, it was just good at it because it wasn't RL'd against it. I know this is a small detail, but it tosses journalistic credibility in my eyes.
> In response to questions about the draft blog post, the company acknowledged training and testing a new model. “We’re developing a general purpose model with meaningful advances in reasoning, coding, and cybersecurity,” an Anthropic spokesperson said. - https://fortune.com/2026/03/26/anthropic-says-testing-mythos...
It is possible it is both, they used to RL against cybersecurity, but also didn't explicitly do any qualitative tests and added/changed more data because of those results. For Mythos, they stopped RL'ing against it, and also now intentionally try to make it better.
Unless of course they've actually noted exactly how things were trained here in some technical report and I've missed it, that's possible. Anthropic aren't famous for being very public about their internals though, but would be curious to read more details about it if it's out there from the horse's mouth.
Presumably RLHF (Reinforcement Learning from Human Feedback).
Honestly though, framing this as a "tech issue" doesn't help IMHO, it just muddies the water. Ever since RSA was invented privacy has been about educating people on how to use it effectively and _why they should care_. If voters now are choosing authoritarianism over democracy and individual freedom, I think we have to face the reality that after almost 50 years of fighting battle after battle on the technology front, we've largely lost the war on the home front in this regard.
I, for one, usually tell my AI to start with secure code, make it small, and modular.
This is the first article I've seen that now says the opposite ! AI will make code too secure!
Since the small amount of AI coding I've done often results in buggy code (even a shell script written today) with the AI go-to solution of "write more buggy code to fix", this seems counterintuitive.
I personally welcome such spiraling offense-defense cycles as it is one of the main drivers of the technological progress.
I remember walking into some shitty congresscritter's office with a fucking years old one pager, with a few more citations written on the bottom in pen because I wasn't going to bother making it pretty this time around.
You should have seen his face when i asked him straight up: dude, you seem to have a problem processing information. Are you having some kind of medical issue? Because I'm not the last staffer: If you abuse my time, I am never coming back here again to add more citations to a fucking one pager from 1999 -- I'm making it my mission to remove you if you fuck this up on purpose ever again.
(Or something to that effect -- I've been told I can get a bit aggressive in my rhetoric.)
This was approximately 2016 and that individual is no longer in office.
I stand by my words.
genai doesn't change that anytime soon?
I thought I've read the worst takes about most things, but this still deserves some kind of prize.
Lmao this is like “I’m concerned the raccoons that I see in the storm drains are going to make our sewer system much too efficient”
Maybe I am overlooking what the author intended but I feel this has already happened and may be the reason why there is so much pressure to create the surveillance state.
I feel more like the push for AI to provide better security (everything?) is an excuse to create a paradigm shift with the ultimate purpose of replacing free societies with fiefdoms of corporate culture returning us to the dark ages but with technology running the show.
It started way before AI. There are good uses of the technology, but obviously we have a culture of criminal accelerationists that are going to use this overpowered tech to shape society into a system of slavery.