AI agent hacks gym to get its user a spot in pilates class
bbc.com
bbc.com
If you ask a judge: probably yes
And publicly disclose that the publicly available software contains the password in plain text could have been construed as aiding a criminal offense.
He was ordered by his customer to look into logging problems.
He found the password in plain text, looked into a database he thought contains only data of his customer and found it‘s data of other customers too.
Any person who, with the intention of securing an unlawful gain for themselves or another obtains for themselves or another data that are stored or transmitted electronically or in some similar manner and which are not intended for them and have been specially secured to prevent their access shall be liable to a custodial sentence not exceeding five years or to a monetary penalty.
If someone hits an unsecured API, receives information, and notifies the company of this while also requesting a bounty, would that satisfy all of the requirements of prosecution?
The unlawful gain is the sticking point in my mind.
That being said using an API does require a minimum of computer knowledge
"But officer, I only walked into that woman's house as an act in itself"
Or life. Where I live it is legal to shoot a person who is an active and immediate threat to your property, self, or other persons. Any person walking through the front yard in the daytime while you are inside, not an active threat. An unknown person entering your dwelling through an unlocked open door is an active threat. In the above case of a woman's house she would probably be praised as a hero, or at least a role model, for shooting the intruder.
PS: in some countries
But say that the underlying api exposes some endpoint discoverability capabilities (eg. Exposing an openAPI spec), then arguably the action was invited: the actions was documented along with the auth model.
We have laws, and LLMs should NEVER break them unless the user states its fine with some qualifying condition. Yes, every country has a different legal system, but I think there's a decent idea of what constitutes intrusion thats agreed on in most parts of the world.
In grey area scenarios, the user should be able to override this, with a warning of clear consequences of, and should they accept, users should be held criminally liable.
If the LLM does so unprompted, the responsibility should be the providers'.
LLMs should be considered tools, legally speaking.
> the bot explained that it had manipulated the system to book him onto classes months in advance - against the normal rules of the system.
> The agent replied saying it had succeeded by cancelling another gym-goer's booking.
This wasn't just breaking the rules of the gym, it also caused damages to someone. Someone had their booking canceled, maybe lost some money, definitely lost time dealing with this topic.
I disagree. If an endpoint is open, my assumption is that it's free for me to call, the same way I expect to be able to purchase items on the shelf at the store. If you fail to clearly label or separate them, that's on you.
The famous "innocent as long as their assumption was" defense. "If the window is open, my assumption is" works the same way.
The law is clear on this, no need to make assumptions. The guy knew immediately the agent broke the rules. A modicum of common sense tells you that those actions can't possibly be above board. The gym will probably let him off the hook and not press charges but not because he was entitled to do it.
> the same way I expect to be able to purchase items on the shelf at the store
I'll try to make my point using the same logic. Why "purchase"? Each item isn't clearly labeled as "must be purchased before exiting the store". And why just from the shelf? Why not the shelf itself? Why not "items from a desk drawer in a back room of the store", or even "items from the pocket of a cashier"? They're all right there, unlocked, and unlabeled.
> If you fail to clearly label
Do you carry a visible label with things people are not allowed to do to you? Without that label do others often make the same assumption you do? :)
It's not straightforward: the conviction was eventually vacated (without really addressing the substantive point), and it is possible that the US authorities went particularly heavy in this case for other reasons.
But yes, attacking an unauthenticated API has previously met the threshold for conviction.
That wouldn't make it remotely acceptable of course.
> (a) Whoever
> (2) intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains—
> (C) information from any protected computer;
> (6) the term “exceeds authorized access” means to access a computer with authorization and to use such access to obtain or alter information in the computer that the accesser is not entitled so to obtain or alter;
A "protected computer" is:
> (B) which is used in or affecting interstate or foreign commerce or communication, including a computer located outside the United States that is used in a manner that affects interstate or foreign commerce or communication of the United States
So this is a Pilates computer, which probably is used by a business which probably conducts interstate commerce, which probably makes it a protected computer, and the API gave "information".
So, yes.
In the original sense of the word, i.e. creative use of technology? Debatable, and most techies would probably land on the "no" side of the debate.
Is it illegal? Depends on the jurisdiction and is up to a judge, but it probably falls under unauthorized modification of computer systems, so here the answer is probably "yes". IANAL, but lack of authorization mechanism is probably not a valid defense. AFAIK lawyers and judges do not follow the law to the letter independent of real world experience, so you can probably argue that if the UI didn't provide a way to cancel other people's appointments, then any reasonable person should realize it wasn't intended for public use and thus refrain from doing it. (Corollary: LLMs are not reasonable persons.) Just like an unlocked house door isn't an invitation to come in, you have to assume it was done accidentally (may depend on customs of the location, but where I come from it's pretty obvious).
Is it morally acceptable? Pretty clearly no. Even if it wasn't illegal, it's a jerk move.
I am off to DeepSeek Flash now.
I use a combination of big American AI and self hosted but I don’t understand your point.
Just without the skin in the game.
...And with a fully grown adult requesting it to do so with zero care or due diligence. Just a whoopsie after the fact.
Why would they let media take photos of them?? I suppose as a "look this could've happened to anyone"? But it actually can't.
I guess they also must've realized that, given that the bbc interview was declined. So good on them I guess.
I'm unclear what you consider reasonable due diligence in this case. Should the protagonist have assumed that the gym API was unauthenticated, and that the agent would discover this?
They should not have given a next token predictor unsupervised Internet access, regardless of what the marketing might have told them about its capabilities or alignment.
Don't use technology you do not understand in ways that might bring harm to others. Exercise appropriate caution.
I don't think the guy itself is fully to blame though. Our whole culture has rotted quite significantly.
Yes.
> regardless of what the marketing might have told them about its capabilities or alignment
I think that marketing teams should be blamed for lies and exaggerations much more then people who trusted them. As is now, everything is too much biased toward the biggest liar out there.
Might "harm innovation" though.
As programmers, we damn well have responsibility for that which we write and execute. AI cannot be treated any other way. And that doesn't mean cutting corporate structures a check to pressure engineers as accountability sinks either. There needs to be a formally recognizable standing for a practitioner to tell their employer "no" that is recognized and backed by the public. Otherwise degenerate/harmful use cases cannot be professionally prevented. There'll always be a more desperate developer willing to take the risk.