The problem is that no one has been able to prove that it is actually worth the cost. That is a very fragile assumption.
Normally, you secure things up to minimize (${cost of security measures} + ${expected damage from attacks that materialized}), writing off actual material damage with insurance wherever possible. You pick security measures based on their effectiveness, which usually translates to "how expensive will it make success for attackers", aiming to push that above the value the attackers can expect to gain.
There are obvious exceptions to that, like risk to life and limb, as well as some other special situations where attackers may have unusual motivations and thus the economic logic of "make stealing treasure cost more than the treasure" stops applying. But those are exceptions. Almost everything you deal with in your life - from your bike shed to the corporation that owns your bank - follows the above logic in terms of security.
--
I spell this out because I've noticed that tech industry circles have this weird, belief in security as some kind of binary, holy good, that you either have and are blessed, or don't and sin. This obsession starts with failing to even recognize, much less ask, the most important questions about security: why do you want to protect it, and who are you protecting it from?
Whether you agree that the potential harms outweigh the benefits in this case or not those calculations are always happening, so yes, I guess you're right. That is society in a nutshell.
You're paying for their services to collect reward for yourself, but also deciding your own risk/reward when choosing e.g. how much access to grant Claude for any given task.
I guess there's the case where the more capable Claude is, the more someone else can use it to find vulns in your services while Anthropic collects their subscription money? But that is mitigable risk that you shipped regardless of what Anthropic is doing.
We haven't yet written those laws for "AI."
I guess it’s the fact that Anthropic usually frame this around morality and risk to society that makes it different. Instead of “risk/harm to me vs reward to me,” their framing reads as “risk/harm to us vs reward to me” or “immorality vs reward to me.” That’s what makes it feel like a great metaphor.
The standard cost benefit analysis we all do justifies increasing the harm to others if the opportunity to benefit ourselves goes up.
At big scale, a single big accident poses a risk to ruin your big operations.
At 1000, the number of fried boards will be more predictable and therefore the risk to the business is lower, even if the long-run averages are the same.
At 100 000, you can afford a better and continuously improving process, and dedicated facilities, and skilled experts, and parts get even cheaper because you're a volume buyer or perhaps own the supply side, and you get to set your own risk curve.
Lots of things get cheaper at scale. Insurance, too.
For my own use of LLMs, I do try to avoid anything which I know has a risk the artefacts they produce may end up DoSing or spamming, and I've avoided the OpenClaw-type pattern for a broader range of reasons of which this is simply one tiny part, but I'm not absolutely confident I could avoid this even in the code coming out of the free tier of the web chat interfaces except by checking every single line of output every single time.
I was thinking you can't make the chance of catastrophic failure zero (we still hear about "Claude deleted my home folder"), but you can definitely limit the blast radius.
You can't get the risk to zero. But the opportunity cost of not playing the game is rising. So you accept some level of risk.
My personal take here is "why screw around with containers and virtualization when a used ThinkPad is $50". Just give it its own machine. Then it can blow it up all it wants. (Or a $3 VPS, as the case may be :)
[0] The lethal trifecta for AI agents: private data, untrusted content, and external communication - https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
Just make sure it doesn’t have ssh access to any other machines!
Technically a merchant could require meeting in person to exchange a OTP to avoid this and make it 0 but it is not worth it and you will get out competed by other businesses willing to take on a marginally higher amount of risk to unlock a lot of utility for the user.
The opportunity cost of not using OpenClaw? I don't think it's that foundational yet that there is an opportunity cost to not using it. Most people have no purpose for a general-purpose AI both in their personal lives and at work, there is no sense trying out OpenClaw when you don't even know what it'll do.
Silently corrupting files, that goes undiscovered until after backup window closes, and data exfiltration are the immediate, serious risks.
Neocon society. Socialism is not like that.
Socialism is not perfect, it is just better than any other alternative.
My answer to the trolley problem is that you're allowed to not kill... unless you're the railway manager. If you're in a position of authority you pull the shit out of that switch, and then drink yourself to sleep at night. This is what authority means, not choosing the "feel good, ignore the people that could have been saved" path.