Hacker News Reader

Comment by morsecodist | Hacker News Reader

This is honestly so interesting to me because I feel like I have the exact opposite experience. I have never experienced the level of hype there is for LLMs both here on HN and in society at large. I see people talking about how amazing they are, how they will replace everyone in a matter of years and even achieve super intelligence. I see people argue strongly against even the mildest LLM skepticism. I use these tools every day because they are useful but I'm also blown away by how often the answers are completely wrong, like not even close. I sometimes feel like I am crazy or everyone is using an entirely different product to me.

But the fact that you feel the exact opposite shows that maybe this is just an artifact of cognitive bias.

View on HN

I guess we are two then, I feel the pretty much the same way as you. I might be completely wrong about all this, but it is getting to a point where it feels like I'm getting gaslit when I compare it with my own experience with these tools so far.

Genuine question as I definitely need more to balance my own thinking —

What has your experience with GPT been? For me, GPT-3 was not really useful as a software dev.

But GPT-4 is miles ahead of that. It’s helped me write code maybe 4-8x faster than usual, and has even allowed me to debug existing issues far, far quicker and more accurately than I’d ever be able to on my own.

Part of the gap very well might be my own mediocrity with development . I wouldn’t argue that folks with far superior skills and novel challenges day-to-day might be unimpressed.

But as an average dev writing pretty boring code (REST APIs and system integration mostly), I’ve been blown away by GPT-4. I am pretty well compensated and have been in the field for 10 years, too; but I am aware of my own shortcomings.

Crafting good prompts makes a big difference. Also, after it gives you an answer, say you think there's an error and ask it to check its own result. It will often correct itself. This self-reflection ability leading to more accurate outputs was pointed out in the new Reflexion paper:

https://nanothoughts.substack.com/p/reflecting-on-reflexion

https://twitter.com/Orwelian84/status/1639859947948363777

Here's another good thread about it:

https://twitter.com/ericjang11/status/1639882111338573824

Thanks for the links, appreciate it!

I've rarely seen completely wrong answers. When I did see them it was because I was too prescriptive.

Like, I might ask "using this library, implement that feature" in the hopes that there it has learned of some way to do a thing I haven't been able to figure out. In those cases I see it hallucinate, which I assume means it's just combining information from multiple distinct environments.

If I'm not too specific, it does a pretty good job.

IMO its biggest fault is that it is not good at admitting it doesn't know something. If they can crank up the minimum confidence values (or whatever, the values used to guess the next token), maybe we'll see better results.

> But the fact that you feel the exact opposite shows that maybe this is just an artifact of cognitive bias.

Or social media/advertising engagement algorithms doing their work. After all, you get more engagement with negative emotions than positive ones.

It seems plausible that you and the OP are at least slightly on different sides of the LLM issue, and so you and the OP could literally be seeing two different realities crafted by engagement algorithms, because it detects that each of you pay more attention to the other side that you disagree with, and that snowballs into seeing only the other side and thinking you're taking crazy pills.

It depends on which one you use. Bard for example is kind of crap.

Reply on news.ycombinator.com