Hacker News Reader

Comment by wodenokoto | Hacker News Reader

I thought the leaderboard was dominated by automated GPT-chat bots.

Did they get banned or couldn’t they keep up with the puzzles?

After about day two, ChatGPT didn't produce anything useful anymore iirc; at least when asked to write the solution in Haskell.

Interestingly, it still produced internally consistent solutions with valid Haskell code, they just didn't have much to do with the given problem. And I feel like that sums up one of the main issues with ChatGPT quite well: if you scrolled through StackOverflow and saw a really polished solution for something, you'd assume it was mostly correct, because for people, 'visible effort' put into code usually correlates with correctness/ourselves. But ChatGPT always produces really polished solutions really confidently, even if they are complete nonsense.

I’ve had that signaling discussion here a few times. With a person, usually they’ll give you some indication (explicitly or not) that they’re uncertain of the correct answer, or they’ll have a relevant track record to help you evaluate their honesty.

No such luck with ChatGPT.

People are confidentally wrong all the time, and ChatGPT does have a relevant track record to help evaluate whether it’s apparent confidence is justified (at that track record, as is true for many people, is “highly mixed, so you shouldn’t implicitly trust it”.)

People were asking chatgpt multiple times, and then taking the most common working answer. Guess that's kind of a method to estimate it's confidence.

> if you scrolled through StackOverflow and saw a really polished solution for something, you’d assume it was mostly correct

IME, that should, at best, be a very tentative assumption for StackOverflow.

Heck, for me it is also a very tentative assumption when I see a very polished solution in official documentation from lots of providers (AWS, I’m looking at you.) Though in that case I assume it was correct at some point in time, for some (possibly not publicly released) version of the software/service.

Yup, in other domains ChatGPT will confidently give answers that sound vaguely plausible but miss out key things a human would definitely highlight, or just get the specifics wrong.

There's a video of Harstem I think, following build orders ChatGPT came up with for Starcraft II. ChatGPT doesn't say "I have no idea how to play this game" instead it confidently offers advice that is... not good. Harstem wins at least one game anyway, but you know, not from the great advice just from being a good player.

GPT only got a good placement on Day 3 Part 1. It seems to not have been very useful in any other days (except for people maybe using Copilot).

Reply on news.ycombinator.com