Hacker News Reader

Comment by ordersofmag | Hacker News Reader

Tell me about the techniques you use to ensure all the code you use is 'correct'. and then explain why those techniques can't also be used by an AI.

View on HN

I read and understand the code using my brain, by constructing a mental model and reasoning about it. An AI can't do this because they don't have mental models and don't do reasoning.

I am out of my depth here and don’t know anything about how other people reason and construct mental models, but I mostly talk to myself about the problem and then do something at the end of that. There’s no point where I like have the whole solution in my minds eye (for programming topics, maybe for a drawing or a sculpture or removing a transmission or something I can do that).

Following the output of agent “thinking” simulation lines up pretty good with what I’ve been doing for 20ish years, but of course I may just be a moron who isn’t good at computers.

I don't think I would conclude you're a moron. I don't think you really "hold everything in your mind's eye" when you understand something. I don't know how I do it or how it really works (nobody does). But the fact that the agent logs resemble how I might talk while thinking doesn't really give me much confidence (since chatbots can resemble all kinds of behaviour they're not actually doing).

There is simple correctness but there are also second order effects to consider. How does this particular implementation allow you to grow, and in which directions? What does it prevent? If you don't already have an opinion about this, then the LLM is going to do something and you're going to have to live with it, because it has no idea that it is "making a decision". And now, neither do you!

This is why LLMs do their best work at "leaf nodes", building on existing infrastructure but not designing new patterns on their own.

LLMs can't introspect, reason, or build internal models of the world. You can get very far without that, but there are some subtle ways it will bite you, and it's a fundamental limitation. Hallucinations are one: they are the feature, not a bug.

Reply on news.ycombinator.com