Comment by bsenftner | Hacker News Reader

Comment by bsenftner | Hacker News Reader

Parent Full thread

A major problem with LLM AIs is their core nature is not understood by the vast majority of everyone - developers included. They are an embodiment of literature, and if that confuses you you're probably operating on an incorrect definition of them.

I like to think of them as idiot savants with exponential more savant than your typical fictional idiot savant. They pivot on every word you use, each word in your series activating areas of training knowledge, until your prompt completes and then the LLM is logically located at some biased perspective of the topic you seek (if your wording was not vague and using implied references). Few seem to realize there is no "one topic" for each topic an LLM knows, there are numerous perspectives on every topic. Those perspectives reflect the reason one person/group is using that topic, and their technical seriousness within that topic. How you word your prompts dictates which of these perspectives your ultimate answer is generated.

When people say their use of AI reflects a mid level understanding of whatever they prompted, that is because the prompt is worded with the language used by "mid level understanding persons". If you want the LLM to respond with expert guidance, you have to prompt it using the same language and terms that the expert you want would use. That is how you activate their area of training to generate a response from them.

This goes further when using coding AI. If your code has the coding structure of a mid level developer, that causes a strong preference for mid level developer guidance - because that is relevant to your code structure. It requires a well written prompt using PhD/Professorial terminology in computer science to operate with a mid level code base and then get advice that would improve that code above it's mid level architecture.

In two words "book smart".

In more words, "of course it's stupid, it's as complex as a mid-sized rodent where we taught it purely by selective breeding on getting answers right while carefully preventing any mutations which made their brains any bigger".

It's basically plotting the dots of all easily accessible written word to find your words, finding the words that are answers to your words and then charting a line through them no matter how scattered those points may be, and spitting that back out. It doesn't "know" anything nor is it reasoning even if the results are similar.

You have to come into it with the same "these people are only stupid and lack the experience to answer my questions despite thinking they do, they lack the world view to even process how I arrived at the parameters of my question(s)" apprehension like you would if asking reddit about some hazardous thing that would make them all screech. AI is the margarine to that butter.

It's a technology with potential to deliver great value, but there are limitations...

Not to put too fine a point on your metaphor, but the different training methods deployed by ChatGPT vs Claude, for example, changes that a bit regarding who did the “selective breeding”, arguably nurture vs nature, respectively

Therein lies the argument that SWEs could become operators (in a much reduced capacity) between AIs and the world.

Isn't this where the research plan implement loop comes in though? Assuming comprehension used effectively?

You should be learning alongside the llm through the research phase of anything. Updating your understanding of what is possible and best practices with rigorous checks and limiting scope to a high fidelity to leave little room for doubt. In-line commenting and questioning and asking for more passes on the living document of the area you are working on and then judiciously breaking it down further when you think there is too broad a scope for an llm to understand and synthesise properly.

If you do end up with too much vagueness, you need to limit scope more or break up the feature, implementation etc to be specific and applies enough to again, properly research and decide the plan.

I guess this is not so easy because lot of it depends on your own ability of reading comprehension, but I've had great success learning niche topics because I research (as a sub agent usually) essentially any topic that is mysterious until every level of the puzzle is properly mapped out to the specificity required.

Do I think most people are doing this? No. So I guess the statistics make sense. It's not intuitive to many people I think - because as you said, it's an embodiment of literature that is a tangled web of thought patterns and perspectives, so you need to pare it's answers down to the specific level, direction and area of ideas you want to get out of it. Way easier to do than it sounds, but it requires finesse in comprehension rather than getting lazy with it - normalcy of deviance comes to mind.

Reply on news.ycombinator.com