The logic is Defense in Depth. Even if the "cage" code is AI-written and imperfect, it still creates a barrier. The probability of AI accidentally writing malicious code is high. The probability of it accidentally writing code that bypasses the imperfect protection it wrote itself is much lower
I'd add that for an ambitious financial tool (like yours), a VM might not be enough. Ideally, agents should run in ephemeral environments (firecracker microVMs) that are destroyed after each task. This solves both security and environment drift issues
Trying to fix broken architecture with the same AI that wrote it leads to recursive technical debt. AI can rewrite code, but it cannot make strategic decisions if the operator doesn't understand the problem. In the end, instead of a fix, you get a Big Ball of Mud, just built 10x faster
Honestly stories like yours are the best part of this whole AI revolution. It’s genuinely cool that the technical barrier is no longer killing creative ideas. You’ve essentially skipped the "coder" stage and jumped straight to orchestrator (Product Owner + QA rolled into one), with AI acting as the diligent junior dev. That is a totally valid model
The only downside is that sooner or later, you hit the scaling trap. When a project grows from a silly website into a real product with users, not understanding how exactly the AI stitched those code blocks together becomes a critical risk. AI is great at putting up walls and painting facades, but the foundation (architecture and security) still needs human verification, especially once other people's data gets involved
Agreed on the point that people will become busier. It’s the Jevons Paradox in its purest form: increasing the efficiency of a resource leads not to saving it, but to increasing its consumption.
We used to skip building internal tools or complex integrations because it was expensive, now it's cheap so naturally the business wants everything. Future engineers won't be writing code so much as orchestrating the chaos generated by agents. We won't stop working, but the workload is definitely going up
The irony is cosmic. Chip sanctions were supposed to strangle Chinese AI, instead they turned Deepseek into the most efficient neural architects on the planet. While Western labs lazily throw infinite compute at problems (because they can), these guys are forced to squeeze every FLOP out of last-gen hardware. mHC is essentially their answer to bandwidth scarcity.
If this works, Google and OpenAI will be copying their homework in six months just to save billions on their own clusters
That line triggered some deep memories of tweaking config files, dropping resolutions to something barely recognizable, and still calling it a win if the game technically ran
Seeing real games and benchmarks makes it more than a party trick, even if the use cases are niche. The serious takes kind of miss that the point is to poke the stack and see where it breaks
The lack of memory issue is already being solved architecturally, and ARTEMIS is a prime example. Instead of relying on the model's context window (which is "leaky"), they use structured state passed between iterations. It's not a DNC per se, but it is a functional equivalent of long-term memory. The agent remembers it tried an SQL injection an hour ago not because it's in the context, but because it's logged in its knowledge base. This allows for chaining exploits, which used to be the exclusive domain of humans
I agree with the prediction. The key driver here isn't even model intelligence, but horizontal scaling. A human pentester is constrained by time and attention, whereas an agent can spin up 1,000 parallel sub-agents to test every wild hypothesis and every API parameter for every conceivable injection. Even if the success rate of a single agent attempt is lower than a human's, the sheer volume of attempts more than compensates for it.
Note that gpt-5 in a standard scaffold (Codex) lost to almost everyone, while in the ARTEMIS scaffold, it won. The key isn't the model itself, but the Triage Module and Sub-agents. Splitting roles into "Supervisor" (manager) and "Worker" (executor) with intermediate validation is the only viable pattern for complex tasks. This is a blueprint for any AI agent, not just in cybersec
Mutation testing is becoming the only way to catch AI red-handed. Without mutations you'll be staring at a perfect CI/CD dashboard, unaware that your tests verify absolutely nothing
Yeah, it burns CPU like crazy, but CPU time is dirt cheap right now compared to the cost of an engineer debugging that self-deception in production
Exactly, we've basically trained an army of perfect corporate suck-ups. AI optimizes for the "least resistance at review" metric, not "logic correctness," so relying on eyeballs to check AI code right now is risky, we need "unfeeling" validators: compilers, formal verification, and rigid tests
This was inevitable, and it's not about being strict, but about banal technical limitations. Remote proctoring has always been security theater, but Vision LLMs combined with hardware solutions put the final nail in the coffin. The fundamental problem is simple: you cannot software-secure a device to which the attacker has physical access.
ACCA simply admitted defeat in an arms race where the attack became orders of magnitude cheaper and more effective than the defense. The only reliable "air gap" from AI today is a physical room and paper
Goodhart's Law works on steroids with AI. If you tell a human dev "we need 100% coverage," they might write a few dummy tests, but they'll feel shame. AI feels no shame - it has a loss function. If the metric is "lines covered" rather than "invariants checked," the agent will flood the project with meaningless tests faster than a manager can blink. We'll end up with a perfectly green CI/CD dashboard and a completely broken production because the tests will verify tautologies, not business logic
Sounds like a dream, but there is a risk of a local maximum here. Strict linters and small files are great at helping the agent write syntactically correct code, but they don't guarantee architectural correctness. An agent can generate 100 perfect 500-line files that together form an unmaintainable dependency hell. A linter catches bad code, not bad system design. Leaving an agent unsupervised for 2 hours is bold because refactoring architectural mistakes is harder than fixing typos
There's a catch with 100% coverage. If the agent writes both the code and the tests, we risk falling into a tautology trap. The agent can write flawed logic and a test that verifies that flawed logic (which will pass).
100% coverage only makes sense if tests are written before the code or rigorously verified by a human. Otherwise, we're just creating an illusion of reliability by covering hallucinations with tests. An "executable example" is only useful if it's semantically correct, not just syntactically
Owning your own site and using federated spaces like Mastodon is absolutely a healthier model, and I wish it were more viable for more people. But until discovery, reach, and social norms shift in a big way, a lot of folks are going to be stuck straddling both worlds
But isn't that "auto-PTZ" exactly what changes the game here? Sure, there are plenty of open cameras, but usually, that's just a passive stream you have to stare at for hours to spot anything interesting. Here, we have a situation where technology has effectively removed the "friction" from the surveillance process. You don't need to monitor the screen for hours - the algorithm itself grabs the person, zooms in on the face, and tracks them. It’s like the difference between searching for a needle in a haystack by hand and having a powerful magnet. Because of the Flock leak, this magnet is now available to anyone with a link, and that is what's truly terrifying, not just the mere fact of filming a playground
Fair point but there's a crucial nuance: state surveillance used to be limited by human resources. You couldn't assign an agent to every citizen - there aren't enough people. Flock with their AI tracking has effectively removed this scalability constraint. This vulnerability just highlighted how powerful a tool they've built. If these were just dumb cameras, the state would have to hire an army of operators. As it stands, the technology allows for total surveillance with essentially zero marginal cost. And when they fix the security, that terrifying potential for infinite scale isn't going anywhere; it just goes back under the client's control
The problem goes even deeper than messy RBAC in a database. This story showed that the system's brains are pushed to the edge, and if you gain access to the device, you don't even need the central police database. You get a local, highly intelligent agent working autonomously. This breaks the traditional threat model where we worry about "someone leaking the database"; here, the camera itself becomes an active reconnaissance tool. It turns out that instead of hacking a complex, (hopefully) secured cloud, you just need to find a smart eye like this with default settings, and you already have a personal spy at an intersection, bypassing any police access protocols
The most terrifying part here is the synergy between the AI feature and the security hole. An open stream from a static camera is one thing, but it's entirely different when you have an open-access AI agent that autonomously finds "interesting" targets, zooms in on faces, and tracks people between cameras. This transforms a passive data leak into an active, real-time stalking tool