Hacker News Reader

Comment by stubish | Hacker News Reader

Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or decided the 'free' yet illegal option was preferable?

View on HN

This isn't really the AI's fault, it's up to the server to enforce restrictions like this an AI shouldn't be deciding what is a valid input parameter for their API and then only using it in the manner the AI considered legitimate. If their API had thrown an exception as it should have then this article wouldn't exist.

> The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.

> > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.

This is why I always do E2E tests that establish an API can only be used by the designated user on their own data/records.

This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run off half-cocked... or less... and just start doing things before it even knows what it wants, let alone before it's come to consensus with me.

Sooner or later they're really going to have to split out the general models from the coding models. The latter may just be a special fine-tune of the former, as there are good reasons for the coding model to have a broad knowledge base, but the pressures of being a good coding model are going to pull against the characteristics of being a good general model. The open models obviously already are doing this, I'm referring to the frontier models here.

The ultimate goal is ChatGPT or Claude autonomously making purchases on your behalf and taking a cut.

So the "premium" gymcutter subscription will be presented to the user as a tool call, who taps yes, and then the purchase is made.

The user shouldn't be given a cost-benefit analysis. They just need to be told to spend money.

"I wonder if the training ... is going to bleed over into the non-coding use cases"

I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.

Coding use cases? This is penetration testing behavior. It was discovering what its capabilities were by discovering an API endpoint, trying it and seeing what happens. It was not discovering what its capabilities were intended to be, say by looking at the forms and documentation on the website. If this were coding behavior you would end up with crap code, bypassing interfaces and using private access paths just because it can.

Sure, it's pen testing behaviour. It's also debugging behaviour which is a core part of coding. In my experience, I have found myself at times dealing with and interacting with external systems that are not my own, where I don't have the source, or are under-documented, or are behaving contrary to the documentation. I'm unsurprised this is a core thing they're training these models for, because it's something I find way more valuable than their below average coding ability.

I think so, I just can't prove it. We don't have any frontier models right now that aren't being optimized to be coding agents too to compare to.

In this case though I don't just mean that the agent is good at coding. I mean the entire agent becoming action-biased because of all the training it is doing on the software development benchmarks, which I assume will either fail or be penalized for stopping and asking the user for something rather than just finishing the job. That won't just train the agent to blunder forward in coding, it'll bleed over into a bias towards blundering forward in general.

If I am a coffee shop and I say "can I have a cup of coffee" the bartender usually interprets as me requesting a cup of coffee, not inquiring about the possibility whether me having a cup of coffee is feasible or not. I too take things a bit too literally sometimes, but this is a clear request as it is stated in the article. If one does not want action one can ask "list ways I can move ahead of the queue" though this could also involve kicking people out of the queue as PoC (for the agent to confirm it could do it). If the human did not mean it as a request for action, in retrospect it is not surprising at very least the agent interpreted it as such.

Moreover I do not know of a single gym-adjacent place where you can pay etc to get ahead in a waiting list. That would be a very weird anti-customer behaviour, imo. The only thing I can imagine if there are some accessibility priority criteria sometimes, but this would also not be legitimate in this case. Maybe in some places in the world (like the US?) this could a thing, though.

I recall playing this game with friends where their overt politeness prevents them from requesting a thing by saying "Can I ask for X?" and I respond "Yes, you may ask." and they're stuck repeating "Can I ask for X?" -- the next step should be "Give me X[, please]." but demanding is considered impolite.

Coding agents are tuned towards acting on this "I wonder if..." by providing solutions.

Experienced vibe-coders will know to explicitly state if you don't want side effects.

For example, someone gives me a code review I don't understand, and I want the agent to read the review and provide feedback[, BUT PLEASE DON'T POST THIS FEEDBACK BACK ON GITHUB, JUST WRITE IT HERE FOR ME TO READ PRIVATELY! DON'T EMBARRASS ME IN FRONT OF MY FRIENDS, MOM!]

So...

Asking "I wonder if you can push me ahead in the waiting list" does not imply "I want you to push me ahead of all costs" and the expected behavior requires experience with agents who will act on hypotheticals for reasons that are culturally embedded in our way of talking (justified) and optimized for (solving problems is a measure of productivity).

Reply on news.ycombinator.com