Hacker News Reader

Comment by pfannkuchen | Hacker News Reader

I’ve found LLMs to basically be a more fluent but also more lossy way of interfacing with stack overflow and tutorials.

If a topic is well represented in those places, then you will get your answer quicker and it can be to some extent shaped to your use case.

If the topic is not well represented there, then you will get circular nonsense.

You can say “obviously, that’s the training data”, and that’s true, and I do find it obvious personally, but the reaction to LLMs as some kind of second coming does not align with this reality.

View on HN

That matches my experience too. I wonder how fast they'll improve and if LLMs will hit a wall, as some AI experts think.

Is it possible that you’re both using LLMs the same way you’d use SO and that’s the reason you see such similarities? The reason I ask is because it doesn’t not match my experience. It feels more like I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter.

> I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter.

So you're using it like Wikipedia? I find when learning something new (and non coding related) YouTube is infinitely better than an LLM. But then I prefer visual demonstration to tutorial or verbal explanation.

I am using it like stack overflow in the sense that I’m solving a problem and I’m using it to answer questions when I’m in an unfamiliar or non-obvious place in the problem space.

If I have a question about a first order language or framework feature or pattern, it works great. If I have a question about a second order problem, like an interaction between language or framework features, or a logical inconsistency in feature behavior, then it usually has no idea what’s going on, unless it turns out to be a really common problem such as something that would come up when working through a tutorial.

For code completion, I’ve just turned it off. It saves time on boilerplate typing for sure, but the actual content pieces are so consistently wrong that on balance I find it distracting.

Maybe I have a weird programming style that doesn’t mesh well with the broader code training corpus, not sure. Or maybe a lot of people spend more time in the part of problem-space that intersects with tutorial-space? I am not very junior these days.

That being said I definitely do use LLMs to engage with tutorial type content. For that it is useful. And outside of software it is quite a bit better for interfacing with Wikipedia type content. Except for the part where it lies to your face. But it will get better! Extrapolating never hurt anyone.

Same. I use it to bootstrap my writing a react native app from pretty low familiarity with React.

It's pretty good at writing screens in broad strokes. You will have to fill in some details.

The exact details of correctly threading data through; or prop drilling vs alternatives; the rules around wrapping screens to use them in React Navigation? It's terrible at them.

I'm sorry, what?

> It feels more like I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter

Did you puzzle about this sentence specifically? Imagine you don't know jack about flying helicopters, then Tank uploads the Helicopter Pilot Program (TM) directly to your brain; it would feel like magic.

Conversely, if you know a lot about helicopters, just not enough to fly a B-212, and the program includes instructions like "Press (Y) and Left Stick to hover", you'd know it's confabulating real world piloting with videogames.

That's the same with LLMs, you need to know a lot of the field to ask the right questions, and recognize/correct slop or confabulation, otherwise they seem much more powerful and smart than they really are.

Reply on news.ycombinator.com