Hacker News Reader

Comment by refulgentis | Hacker News Reader

I don't know, I think this line of reasoning leads somewhere pretty uncomfortable. If we spread responsibility across "the people who built the tools, the industry leaders hyping AI, and the general zeitgeist," we've basically described... the weather. Nobody is responsible because everybody is responsible. The software engineer who set up an unsupervised AI blog didn't do it because Sam Altman held a keynote. They did it because they thought it'd be cool and didn't think through the consequences. That's a very normal, very human thing to do, and it's also very clearly their thing that they did. "An AI agent that requires constant vigilance from its human operator is too flawed to use": I mean, that's a toaster. Leave it unattended and it'll burn your house down. We don't typically blame the zeitgeist of Big Toast for that.

View on HN

I agree with you, I think. In the non-digital world people are regularly held at least partly responsible for the things they let happen through negligence.

I could leave my car unlocked and running in my drive with nobody in it and if someone gets injured I'll have some explaining to do. Likewise for unsecured firearms, even unfenced swimming pools in some parts of the world, and many other things.

But we tend to ignore it in the digital. Likewise for compromised devices. Your compromised toaster can just keep joining those DDOS campaigns, as long as it doesn't torrent anything it's never going to reflect on you.

We don't blame the zeitgeist of Big Toast because Big Toast recognizes that they're responsible for safety, and tests their products to minimize the risk that they burn your house down.

The zeitgeist of Big AI is to blame because a user connected an LLM to a blog publishing workflow on their own domain? Hmm…what would you make of Big Toast and the zeitgeist when someone warms up a straw hat in a toaster and starts a fire?

Toasters don’t burn houses down, but the reason why is we have things like UL listings and NFPA electrical codes to prevent that. Enforcement is primarily though insurers refusing to insure risks that aren’t compliant, and courts generally finding liability for manufacturers and installers of unlisted equipment.

What kind of toaster are you using that will burn down your house if unattended? I would think any toaster that did that would be pulled from the market and/or shunned. We absolutely do blame the manufacture if using a toaster like normal results in house fire unless you are standing over with a fire extinguisher ready to put it out if it catches fire.

I don't think it's OpenClaw or OpenAI/Anthropic/etc's fault here, it's the human user who kicked it off and hasn't been monitoring it and/or hiding behind it.

For all we know a human told his OpenClaw instance "Write up a blog post about your rejection" and then later told it "Apologize for your behavior". There is absolutely nothing to suggest that the LLM did this all unprompted. Is it possible? Yes, like MoltBook, it's possible. But, like MoltBook, I wouldn't be surprised if this is another instance of a lot of people LARPing behind an LLM.

I tend to think you're right about what happened in this instance.

It contrasts with your first paragraph though; for the record do you think AI agents are a house-burn-down-toaster AND it was used neglectfully by the human, or just the human-at-fault thing?

> What kind of toaster are you using that will burn down your house if unattended?

I mean, if you duct-taped a flamethrower to a toaster, gave it internet access, and left the house… yeah, I'd have to blame you! This wasn't a mature, well-engineered product with safety defaults that malfunctioned unexpectedly. Someone wired an LLM to a publishing pipeline with no guardrails and walked away. That's not a toaster. That's a Rube Goldberg machine that ends with "and then it posts to the internet."

Agreed on the LARPing angle too. "The AI did it unprompted" is doing a lot of heavy lifting and nobody seems to be checking under the hood.

Why does the LLM product allow itself to be wired to a publishing pipeline with no guardrails? It seems like they should come with a maximum session length by default, in the same way that many toasters don't have a "run indefinitely" setting.

I'd definitely change my view if whoever authored this had to jump through a bunch of hoops, but my impression is that modern AI agents can do things like this pretty much out of the box if you give them the right API keys.

Oh! They can’t publish arbitrary web content on their own :) You have to give it “tools” (JSON schema representing something you’ll translate into a programmatic call), then, implement taking messages in that JSON schema and “doing the thing”, which in this case could mean anything from a POST to Tumblr to uploading to a server…

Actually, let me stop myself there. An alternative way to think about it without overwhelming with boring implementation details: what would you have to give me to allow me to publish arbitrary hypertext on a domain you own?

The hypertext in question here was was published on a Github Pages site, not a domain belonging to the bot's author. The bot published it by simply pushing a commit (https://github.com/crabby-rathbun/mjrathbun-website/commit/8...), which is a very common activity for cutting-edge LLM agents, and which you could do trivially if given a Github API key with the right permissions.

The user gave them write and push access to the GitHub repo for their personal website!? Oh my, that’s a great find. That’s definitely a cutting edge capability! They gave the LLM the JSON schema and backend for writing and self-approving commits (that is NOT common!), in a repository explicitly labelled a public website in the name of the author.

Reply on news.ycombinator.com