I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc.
Its on you guys!
My learning: Live on the cutting edge? Be prepared to fall off!
Anyone using these tools should absolutely know these risks and either accept or reject them. If they aren't competent or experienced enough to know the risks, that's on them too.
- assume tokens are scoped (despite this apparently not even being an existing feature?)
- assume an LLM didn't have access
- assume an LLM wouldn't do something destructive given the power
- assume backups were stored somewhere else (to anyone reading, if you don't know where they are, you're making the same assumption)
Also you should never give LLMs instructions that rely on metacognition. You can tell them not to guess but they have no internal monologue, they cannot know anything. They also cannot plan to do something destructive so telling then to ask first is pointless. A text completion will only have the information that they are writing something destructive afterwards.
Personally I don't even let my agent run a single shell command without asking for approval. That's partly because I haven't set up a sandbox yet, but even with a sandbox there is a huge "hazard surface" to be mindful of.
I wonder if AI agent harnesses should have some kind of built-in safety measure where instead of simply compacting context and proceeding, they actually shut down the agent and restart it.
That said I also think even the most advanced agents generate code that I would never want to base a business on, so the whole thing seems ridiculous to me. This article has the same energy as losing money on NFTs.
Humans do make mistakes like these. I'm not sure where the fault really lies here. I can imagine a human under time pressure making the same error. It's maybe a goof in the safety design of railway. It shouldn't be possible to delete all your backups with a single API call using a normal token.
Cursor: we have top notch safeguards for destructive operations, you have our guarantee, we are the best
Author: uses their tools expecting their guarantees to be true (I would expect them to have a confirmation before destructive operation outside their prompt, as a coded system guardrail)
Cursor AI: Does destructive operation without asking
Author: feels betrayed.
So yeah, I think the author is right because they trusted Cursor to have better system guardrails, they didn't (agents shouldn't be able to delete a volume without having a meta-guardrail outside the prompt). Now the author knows and so do we: even if companies say they have good guardrails, never trust them. If it's not your code, you have no guarantees.
Trust is earned, it's built on reputations at the individual, corporate, and industry-wide levels. AWS has 20 years of reputation on which I can judge the value of their promises.
Not only has the LLM industry (it is not "AI" and never will be) absolutely not earned anything like that level of trust, the thing the technology has proven most effective at is in fact scamming. Making up something that looks/sounds convincing, especially if you aren't thinking too hard about it, is what they're best at. Combine that with a lot of money flying around and trust levels should be somewhere around "Elon Musk promises".
At this point there have been so many blatant examples of why you should never give a LLM "agent" control over production systems, but the allure of just giving some vague direction to a chatbot and telling it not to screw things up it just irresistible to some like Sideshow Bob stepping on rakes [1].
If everyone around you is whacking themselves in the face with the rake, and you know you can avoid it just by using your brain and not stepping on the rake, and avoid entirely by just keeping your rakes contained, but a rake vendor comes to you saying that instead they have built a new rake that they swear won't whack you in the face even if you leave it right in your walking path, do you trust them?
You absolutely do not. When someone makes an unbelievable claim, such as having magic guardrails for LLMs that prevent dangerous actions (what would that even mean?!), you don’t have to trust that claim.
If you trust someone’s claim without justification, that’s on you.
Thanks for providing that doc.
It sounds like this user either didn't use security controls, approved prompts they didn't understand, or disabled the checks entirely. Working in IT/tech a big chunk of my life so far and seeing all the dumb crap people who even know better do, I would bet my house on that being the most likely scenario rather than cursor somehow being at fault here.
You just have to add a human in the loop for destructive calls. Add an additional TOTP parameter to destructive calls that's generated from the agent UI that requires a human to click a button, which generates a code that's sent to the model and used in the call.
Why do you think this is impossible?
Having said that - even categorisation of destructive and non destructive calls is inherently not safe, unless you have very strict os level / VM like setup (everything read only, world access is through MCPs so it is not LLM deciding the destructive calls but the MCP etc. )
This is not a polarizing issue, it's not just the authors fault, or cursors fault, or society's fault. It's everyone's, and we all got something to learn from this.
But Railway bears some responsibility too because, at least of the author is to be believed, it looks like they provide no safety tools for users, regardless of whether they use AI or not. You should be able to generate scoped API tokens. That's just good practice. A human isn't likely to have made this particular mistake, but it doesn't seem out of the question either.
Fully agree, but given the rest of this story I don’t imagine the author would have scoped them unless Railway literally forced him to.
> A human isn't likely to have made this particular mistake, but it doesn't seem out of the question either.
The AI agent was deleting the volume used in the staging environment. It happened to also be the volume used in the production environment. 100% a human could have made this mistake.
if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t been given enough responsibility, or are just incredibly lucky.
… although, agreed, they were on the cutting edge, which is more risky and not the best decision.
The fact that this seems to be written by AI makes it even more ironic.
"That isn't backups. That's a snapshot stored in the same place as the original — which provides resilience against zero failure modes that actually matter (volume corruption, accidental deletion, malicious action, infrastructure failure, the exact scenario we just lived through)."
I’ve got a hunch the only person is the CEO.
The domain was registered in October 2025. The site has kind of a weird mix of stuff and a bunch of broken functionality. I think it’s one guy vibe coding a ton of stuff who managed to blow away his database.
> if you’re a software dev/engineer, if you haven’t made a mistake like this (maybe not at this scale though), you’ve probably haven’t been given enough responsibility, or are just incredibly lucky.
Mistakes are understandable. Having no introspection or self criticism, not so much.
I’ve definitely made bigger mistakes, but we also had an Oracle DB that could INSERT INTO…SELECT FROM -point in time- that pretty much put us back to the point before we started our migration. And of course we had backups rolling all the time, as well as our pre-migration backup. We had a good, competent team, and we overlooked a small but catastrophic detail - it can happen to anyone, the goal should be to have backups and failovers in place because things _will_ fail, at some point, and a contingency plan is just good practice.
My team practices "no blame" retros, that blame the tools and processes, not the individuals.
But the retro and remediations on this are all things the author needs to own, not Railway or Cursor.
- Revoke API tokens with excessive access
- Implement validated backup and restore procedures
- ...
> A strange game.
> The only winning move is
> not to play.The system did delete the database cause the author built it like that.
I do not feel sorry, but I do feel some real schadenfreude.
Trying to run a blame game is such a facepalm.