Some observations of LLMs that I suspect follow from the way they work and the way they are trained:
- they are amoral
- they have no innate sense of proportion
- they cannot assess their own confidence in-band
Part of the problem with this, I figure, is that the training corpus for code/tech related tasks does not really contain that much discussion about these things; it’s mostly sets of instructions for given tasks, descriptions of exploits etc., so each possible approach leads to other approaches.
There is no easy way for them to learn when they have crossed a line, or when they have gone too far down the rabbit hole, etc.
Useful (arguably essential) for a security analyst, and the tenacity you want from a one-shot demo coder, but for general agentic assistants the industry is going to have to develop some way to manage this sort of extension of trespass.
It often reminds me of Gary McKinnon’s defence, and that of other teenage hackers, which you can reduce to: it was possible so it felt like it was allowed.
This is true of APIs and it is how Silicon Valley has approached disruptive businesses, but it runs up against our cultural notion of “misuse”: uses that are technically possible and shouldn’t be precluded, but are contextually unwelcome because they have undesirable outcomes.
My expectation is that we will lose any sense that misuse is punished or viewed with suspicion or contempt, since that is the rolling trend of the 21st century tech industry. Uber succeeded through misuse.
But the problem is that we will also begin not to be able to punish abuse; if it’s possible to get something by abusing your site/API or by treating your service as an API, then it will become OK, legal and normal for the AI companies to abuse you.
It feels like we are getting there already.