Hacker News Reader

Comment by vishalkundar | Hacker News Reader

The gap between "useful chatbot" and "useful agent" is way bigger than people realize. A chatbot can be wrong 10% of the time and still help you. An agent that's wrong 10% of the time is sending bad emails and making wrong API calls with no one checking.

View on HN

I see this as the gap between an general-purpose agent and a coding agent. A coding agent can imagine something to be true, test it, discover that it's wrong, and recover.

But if you go beyond what can be tested easily, asking the agent to do real work rather than writing a patch, imagining things to be true is a problem.

This to me is the big leap from being good at coding to being good at many other tasks.

Coding could be treated as a low stakes (time & money consequences for retries) closed loop system where most other tasks cannot.

If it screws up booking your flight/hotel room, how does the agent verify this, and even if it verifies.. there is an actual cost to changes/cancellations.

Similar with agentic e-commerce, lots of ability to screw that up and just seems ripe for fraud / being picked off by bad actors.

To reply to myself here..

I can STILL replicate this behavior in Google AI summaries 10% of the time:

"is <SOMEPLANT> ok for cats"

to which it replies: "Yes, <SOMEPLANT LONG SCIENTIFIC NAME VERBOSE PHRASING> is toxic for cats"

The other one going around this weekend: "how long hot dogs on grill"

Summary: "The hot dogs on your grill are likely around 5-6 inches long .. "

So scale this category of error to unsupervised agents with access to your credit card.

This repros nearly 100% of the time on most LLMs, even the most advanced ones: https://share.gemini.google/u9NwYu7lbgxe

n=1 but I gave this to Sonnet 5 medium effort (free model) and it had no trouble with it

Try it without "reasoning". As you can see in my example (and GP), it meanders to correctness eventually after emphatically being wrong, and most reasoning modes hide that from you.

If LLMs worked the way people want to believe they do, there’d be no reason to start in the wrong place — a computer should have the facts!

Seems like to make agents safe we need tentative, reversible transactions. How do you set up a travel plan and then review it? How do you modify it later?

Unfortunately, travel keeps getting less flexible, with worse cancelation policies.

Because a lot of things in the real world require forward planning, I don't think everything can be just-in-time and tentative/reversible. Some things have to be committed to else you have real consequences and lose your money.

There is no way to do cheap and flexible air travel. If the tickets are cheap, than just a few last minute cancellations can destroy your margin for that flight, if you were to offer them for free. Add in the the risk of your seat reservations being gamed by filling up seats and then canceling, and you get a recipe for disaster.

Trains are usually different because they are much cheaper to operate per trip (for the train operator, not the track operators, but that's a different discussion), so running a half empty train is much less of a problem - especially since you don't need to plan for how much fuel to use ahead of time.

Right, these are actual physical real world problems that aren't going away just because it would be easier for agentic workflows.

Another example - agentic food ordering. How much more convenient would this make your life vs how much of an error rate would you tolerate if the cost/repercussions are on you?

Would a customer be happy if 2% of the time it sends 20 pizzas to a random address in their contacts list instead of 2 pizzas to their own home? Or 5% of the time it completely ignores your dietary restrictions/allergies and orders an entire meal of food you explicitly told it you cannot eat?

Real world problems don't go away just because it would make the tech neater & tidier.

It’s an age old control systems problem: open loop vs closed loop

The problem is that with text/code, judgement is hard. Here is what it looks like for physical activity: https://www.youtube.com/shorts/lK7TjujKQLw It's hard to see how that it's not useful at best and could be a disaster for any unsupervised use.

The gulf is bridgeable. The problem is that a lot of people are building agents without strong enough judgment layers around them. Work that can be verified with reasonable accuracy are the sweet spot right now.

How many of these layers are just trying to rediscover/rebuild the idempotence of code?

> The gulf is bridgeable.

Only with an LLM that's actually at agent-quality.

If "useful chatbot" and "useful agent" are two rungs on a ladder, the rung before them is "useful autocomplete". Autocomplete that only gets the next token right 90% of the time won't give you compiling code.

This is much harder than it sounds. Most techniques I’ve seen end up using separate agents to do the planning, implementation, and judging.

The elaborate workarounds you have to build to help an agent which fundamentally doesn’t know what it’s doing reminds me of this old blog post about TDD: https://pindancing.blogspot.com/2009/09/sudoku-in-coders-at-...

IMO present technology is tailored for an experienced developer to give agents manageable tasks that can be one-shot. The marketing right now reminds me of the 90s when AskJeeves promised natural language search when the technology was fundamentally still stuck in keyword search, and learning to craft a search query for Google is today’s prompt engineering

Reply on news.ycombinator.com