Hacker News Reader

Comment by cs702 | Hacker News Reader

Nice!

Briefly, the idea is recursively to decompose tasks into the simplest possible steps, recursively call (relatively small) LLMs as agents to execute one step at a time, and using a clever voting scheme to choose how to execute each step. The authors use this technique to get a relatively small LLM to solve Towers of Hanoi with 20 rings (1M steps). All of it using natural language.

The most obvious question is whether other tasks, more interesting -- less "rote" -- than Towers of Hanoi, can similarly be recursively decomposed into simple steps. I'm not sure that's always possible.

View on HN

This works because a problem could be broken down to a prompt which rarely hallucinates.

Most real world prompts can't be reduced to something so consistent and reliable.

Their key finding was that the number of votes grows linearly with number of prompts you are trying to chain.

However the issue is that the number of votes you need will grow exponentially with hallucination rate.

> into the simplest possible steps, recursively call (relatively small) LLMs as agents to execute one step at a time, and using a clever voting scheme to choose how to execute each step.

It's like humans! Everything old is new again :)

Why not? That's basically how NASA manages large projects.

NASA has done a lot of amazing things but I wouldn’t bet on them winning a Super Bowl.

They'd have a 50% chance of winning one on Mars, since it would just be NASA vs China

Every year NASA has a 50% chance of winning the Superbowl- even on Earth!

Either they win or don't. /s

I could imagine that even a small task at NASA might involve more knowledge and logic than the smallest task for a Hanoi's tower problem.

Depends on what is considered as small enough for the LLM to be resolved with a high confidence.

One issue I often run into with this stuff is the tightly coupled nature of things in the real world. I’ll fashion an example:

Let’s say you break a job down into 3 tasks: A, B and C. Doing one of those tasks is too much for an LLM to accomplish in one turn (this is something you learn intuitively through experience), but an LLM could break each task into 3 subtasks. So you do that, and start by having the LLM break task A into subtasks A1, A2 and A3. And B into B1, B2 and B3. But when you break down task C, the LLM (which needs to start with a fresh context each time since each “breakdown” uses 60-70% of the context) doesn’t know the details of task A, and thus writes a prompt for C1 that is incompatible with “the world where A1 has been completed”.

This sort of “tunnel vision” is currently an issue with scaling 2025 agents. As useful context lengths get longer it’ll get easier, but figuring out how to pack exactly the right info into a context is tough, especially when the tool you’d reach for to automate it (LLMs) are the same tool that suffers from these context limitations.

None of this means big things aren’t possible, just that the fussyness of these systems increases with the size of the task, and that fussyness leads to more requirements of “human review” in the process.

I've been experimenting with this with a custom /plan slash command for claude code, available here: https://github.com/atomCAD/agents

Planning is definitely still something that requires a human in the loop, but I have been able to avoid the problem you are describing. It does require some trickery (not yet represented in the /plan command) when the overall plan exceeds reasonable context window size (~20k tokens). You basically have to start having the AI consider combinatorially many batches of the plan compared with each other, to discover and correct these dependency issues.

>the LLM (which needs to start with a fresh context each time since each “breakdown” uses 60-70% of the context) doesn’t know the details of task A, and thus writes a prompt for C1 that is incompatible with “the world where A1 has been completed”.

Can't that be solved with sub agents? The main agents oversees on combines code and calls sub agents for each tasks.

"basically" is doing a lot of work in this sentence.

Reasoning by analogy is great for intuition, but doesn’t guarantee real results hold. Consider “voltage is like water pressure in pipes, so if there’s a cut in my wire’s insulation, the device won’t get enough voltage” — clearly this is not true, even though it relies on an analogy that’s generally useful.

I really like that analogy, thank you for it. Also applies to “it’s overvoltage, so I just need to poke a little hole in it to let the excess bleed out”…

That one can work, briefly, depending on how conductive your tool is.

Well, corona losses are a thing, after all.

If air was highly conductive that analogy would totally hold.

"If there’s a cut in my wire’s insulation, the device won’t get enough voltage" doesn't follow from: "voltage is like water pressure in pipes"

So I don't really get your point.

> "If there’s a cut in my wire’s insulation, the device won’t get enough voltage" doesn't follow from: "voltage is like water pressure in pipes"

I absolutely agree! In the same way, "an LLM can solve complex problems if it breaks them into subtasks" doesn't follow from "NASA breaks large projects into smaller parts"

This is a really good analogy because the complex intersections between multiple groups independently working and trying to collaborate together into a collaborative hierarchy towards one large goal was one of the things that hid a lot of the problems that led to the Challenger disaster, according to Feynmen.

It is also what made the space shuttle possible in the first place, so I'd be careful about generalizing too much from that observation.

The space shuttle’s design was also deeply flawed to the point it failed to do the core objective, significantly lowering costs. Instead the core mission was sacrificed to meet some arbitrary design goals such as being able to de-orbit heavy objects.

That’s the core issue with decomposition of tasks, you aren’t communicating back up the chain and finding globally optimal solutions unless the task is simple enough to be completely understood.

I'm pretty sure the problem with the shuttle was that it had too many (possibly conflicting) goals instead of one large goal.

It's manned, even though most launches probably could be done without crew. The deadly Challenger launch was risking human crew for something as mundane as launching two satellites into space.

Because it's manned, it has to be able to land at airports, because retrieving astronauts at sea is an unreasonable complication for launching a satellite. Damage to the wings will cause loss of the entire aircraft, something that is unlikely to happen to a capsule.

Because it is a horizontal landing system, the aerodynamics favor putting the shuttle on the same level as the external fuel tank, which exposes the wing to debris from the top of the external fuel tank. If you try building a vertical shuttle in KSP, you will notice that the wings give you too much control authority during launch. Fins are best placed near the bottom of the rocket.

It's reusable, which means wear and tear can secretly accumulate without you noticing. This significantly increases the design requirements for the critical components, like the SRB that had a poor "tang" design, which, as it turns out, was definitively not fit for reuse.

IBM tried that with CMM (capability maturity model), it didn't work, the problem is NASA knows what they're building, rockets and satellites don't have any grey areas and everything is specified. Other things are less well defined, and the people specifying aren't rocket scientists.

Its LLMs all the way down :-)

This can't be scaled to more generalised tasks. If you solve that then you've solved the hallucination issue.

> All of it using natural language.

Combining this with those approaches that recursively reason in latent space would be interesting.

It seems like this could be implemented by any harness.

Reply on news.ycombinator.com