Hacker News Reader

Comment by rfgplk | Hacker News Reader

Frankly, there is no point in trying to "understand" what an LLM does. Their thought process is effectively undecipherable by humans (it's essentially information arising from information) so even such a "simple explanation" is almost certainly wrong. The agents might appear to have "used this method", but the actual method of computation is far beyond our grasp.

Why are people being so belligerent about this? I thought it's fairly obvious at this point that LLM reasoning is far beyond anyones understanding. Or does anyone have a refutation?

View on HN

This is a strange attitude. When an agent is optimizing a piece of code, comes up with 2 variations, and runs benchmarks on them to figure out which one is faster, then selects one of them based on tradeoffs between performance and other things it reasons about, do you ignore its explanation and all experiment runs?

>Their thought process is effectively undecipherable by humans (it's essentially information arising from information

Are you trying to say that human brains are incapable of inference?

You're confusing the weights of a model and internal chain-of-thought with the output of the model. Yes, we don't know a lot about how the internal mechanisms work. But with the correct prompt, agents will produce a worklog that documents exactly what solutions were tried and how the result was obtained.

What are you on about? I have had Fable come up with new shit for me several times (I do research for a living, so actual new shit nobody knew before), and each time it was perfectly understandable.

Of course I don’t know how it got its ideas for what to try. But heck, I don’t even understand how I get my ideas half the time. But the process, like what code it wrote, simulations it ran etc can be understood by (some) humans just fine!

Yes I saw 3Blue1Brown say the same thing in his tutorial on how neural nets worked where he built a simple model to recognize a particular letter. Good reminder.

I've been dabbling with some of my own (tiny) models recently and it's actually shocking at what they can "learn" despite having _zero_ mention of it in it's training data.

Reply on news.ycombinator.com