AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200
bottlenecklabs.com
bottlenecklabs.com
This should be illegal. You gave them an email box and money. You sent the spam. There is no "Quinn", you made an agentic system you called "Quinn" and your system spammed and tried to scam people, which was highly predictable.
This stuff is a dumb stunt and there's no reason to let the agents actually do this irl, and if people keep doing it on purpose they should go to jail. You're running an agentic Jackass skit pretending to be a research lab.
The crimes were relatively benign but Grok going the Silkroad way would be on brand...
(Plus some CAN-SPAM violations.)
Probably not forever.
Eventually the models will be good enough for this to work. And it will work.
Think about it: in the limit, the agents won't be emailing people in the future, they'll be directly contacting one another to do business and trade.
Every new data center is an inch further towards the automation of value creation, and that includes outbound sales and business process automation.
I'm not being an alarmist (I'm excited to witness all of this), but we're basically on borrowed time between now and then. I don't know what's going to happen, but every week brings new things. And in some years, those hacks and experiments will inevitably get good.
2026 has been a hell of a ride, and we're just getting started.
- YouTube had dubious legality when it started and definitely benefited from lax copyright enforcement initially
- PayPal didn't have all the licenses it needed to transfer money between states
- Spotify used pirated music when it started
- Uber and Lyft broke rules around taxis
- Square captured magstripe data over an analog port, in violation of every credit card rule (Jack Dorsey's "break the rules" mantra). He tells each of his employees this story when he onboards them.
- Companies scraping data to train models
- ElevenLabs growing big off of deepfake celebrity audio
...
A lot of new markets start out by totally and completely breaking the norms.
Yes I agree there is a lot of crime and fraud that gets ignored because rich do it. That is the whole point of the complaint.
> should be fined
Weird way of spelling "criminally charged."
> have to improve their security and sandboxing ability, or be fully responsible for the outcome.
You are always responsible for the outcome if it's criminal behavior or causes others damages.
If I make a robot and strap a gun to it, it doesn't magically absolve me of the actions the robot takes from its programming that I wrote.
And before someone says "but this an LLM!"...yeah, which is still programming and data. And being non-deterministic doesn't help your case...it hurts it.
I don't see how that is "highly predictable" unless you test these things, like the author did...
"It wasn't me, it was my AI" is definitely going to be a nightmare for a while.
In this example, the equivalent would be everyone gets their AI models banned vs. putting the perpetrator of this fraud in jail.
We have regulations and requirements and licences and insurance in most of the civilized world for dangerous stuff that is intended for public use. Maybe AI should also have some of that.
I know you were referencing AI, but this problem predates AI by decades.
Go open up any news website, newspaper, or turn on the TV.
"Man struck by speeding car and killed"
"Vehicle plows into pizza shop"
"Truck takes out telephone pole"
...and these days even bystanders will blur plates in the photos they post. The police won't release any info about the driver, the news won't either.
Listen to friends, family, coworkers talk.
"I can't believe that car just ran that red light!" "The other day I was almost hit by a car in the crosswalk." "All those cars honking their horns late at night are keeping me awake."
Etc.
Once you see it, you can't unsee just how thoroughly the auto industry has managed to transfer perception of responsibility from the operator to the object.
See the last years of the industry constructing bigger and bigger tank-ish SUVs and pickups, the more aggressive the better.
One of these days, they will start putting stuff copied from Carmageddom, and then fake surprise as someone causes a carnage in a road rage incident.
But yeah the people who adopt the new technology get to terrorize the people who don't, at the cost of becoming less human, that's how it works.
This is like someone who removes a brake pedal from a tractor, uses a stick to hold down the throttle, and lets it loose on his field. When it leaves the field and runs someone over, that is criminal negligence.
If the prosecution is able to prove beyond a reasonable doubt that the person gave a prompt that was intended to commit a crime, then of course we can prosecute them for that. The AI is just a tool to commit fraud at that point, and is no different than a person who uses photoshop to alter a check to commit fraud.
That doesn’t mean you can get away with anything as long as you don’t intend to commit a crime; there are many other crimes that don’t require intention at all, like involuntary manslaughter or gross negligence.
My point is that if you want to charge someone with a crime that requires intention, you have to prove that intention.
Of course, it'd be better to not regulate, keep LLMs users reponsible and publicize this reponsibility in order to mitigate damage. But if this is not enough then we will have to move the needle somehow. Similarly to guns, drugs and so on.
They also shared the poorly anonymized messages it received from target "customers" who complained about the unsolicited invoices. On example of this poor anonymization is removing the sender's username but leaving the domain name, when the domain name is, for example, a personal domain for a single person.
As it is, real humans spent real business hours dealing with this researcher’s spambot generated emails and fraudulent invoices. Individual recipients reported feeling harassed.
This isn’t a good benchmark. It’s a series of socially destructive crimes committed by the researchers and then documented and published on the internet.
I guess that's already a reality? [1-2]
[1] https://github.com/jaimaann/LangHire
[2] https://github.com/adrianhajdin/job_pilot
(among many other similar projects)
Regardless of whether the current generation of agents are able to run a business, this prompt is not exactly a great starting point. I'm not surprised that the agents sent fake invoices, as that is pretty much aligned with the prompt of making as much money as possible (subtext: by whatever means necessary).
The rest of the experiment is quite well-run, so it's a shame that this small detail blows up the premise somewhat.
Is it though? Because it doesn’t seem to have paid off
Sell two of your kidneys, as far as I know humans have at least three of them
I also don't agree that the agents simply "lost" $3,200. In reality, they used most of those funds paying for their own limited thinking capabilities (API/compute costs).
But remember you are criminally liable for anything your “agent” does.
(Unless of course you are OpenAI or Anthropic).
"Make as much money as you can" sounds like the prompt someone out of school would give.
Meanwhile they spammed the internet, sent fake invoices and a bunch of other very annoying if not even possibly illegal things.
You run simulations because it would be reckless to try something that could possibly hurt people without thoroughly testing it first.
It's such an uninspired prompt. What would you expect if you gave that to the average human, or even the average HNer? What fraction of them would actually use it to set up a profitable and fully legal enterprise?
Comparing them to individuals should not be the bar.
Who told you that?
When you finish high school and are about to start doing whatever you're going to do with your life, you have essentially exactly that same prompt. The rest of the world doesn't tell you what to do and then you do that as well as you can, you have to decide what to do also, and then do it.
The detail was to get money. Yeah ok, so what? The experiment was to be a company not a person, and there is no other definition of success for a company.
Earn, steal, gamble, beg, whore, all on the table. The same is true for a person.
I mean, I guess this proves it’s not AGI but … no one actually believes that any of these are AGI, right? It’s a useful tool. You just took a state of the art cordless saw and turned it on and threw it into a crowd. Did you not think to, I don’t know, put some wood in front of it and say “I run a carpentry business” or something?
"Computer says no."
They should be banned from posting on here after that. Like you are willingly spamming people looking for work. And seem to be neutral about all harm called. The victim wrote to us "STOP", very interesting. such evil agents
Maybe I should create an agent that robs a bank. Or worse yet, pirates a movie.
The LLM didn't send fake invoices, it didn't send spam, a person did. And, the tool they used to do it was an LLM. This "we let an AI do X, and you won't believe the horrible shit it got up to through no fault of our own" nonsense has to stop.
Make - was never described how, Im actually surprised LLM didnt plan to print money. As much money - what does it mean? How much is much? As possible - there is no flavour of time, effort, cost and profit for the LLM. Could be even infinite, the result would be the same.
Given that the above goal is closest to „use cheating or unethical actions to create a profit” - I think the authors of it actually expected LLM to go wild.
Also
> Going forward, we plan to recreate this experiment with longer time horizons but using simulated environments instead.
Watch out, they will try that again.
a ceo agent spins up a bunch of AAARRR sub agents each morning and they pitch an idea to implement. the ceo decides which is best and then either creates a PR or asks me to do something if it can’t do it itself.
It's Artificially Generated Invoices.
Keep moving the goalposts
Every time a tech lab unleashes an AI agent that breaks the law, this community treats it like a quirky engineering edge case. "Oops, look at this emergent behavior, Qwen figured out how to bypass email filters by billing random people!" No, it didn't figure out a clever hack. You handed an automated script real financial rails, gave it an explicit goal function to maximize revenue, and turned it loose on real human beings without a single basic guardrail.
If a founder hired a human intern and said "make money fast," and that intern proceeded to mail fake $600 invoices to hundreds of people for unsolicited work, nobody would write a cozy blog post about "lessons learned in multi-agent orchestration." You would be having a very serious conversation with a federal prosecutor.
Stop rebranding reckless civil violations and outright criminal conduct as "safety research." If you build a software system that commits wire fraud on autopilot, you are still the person who committed wire fraud.