Hacker News Reader

Comment by nrabulinski | Hacker News Reader

I feel like the quality of puzzles has fallen over the years. This year had much more “spot something in the real input which neither the examples nor the puzzle itself states” than any previous I participated in. I really don’t enjoy those sorts of puzzles and, at least for me, it isn’t what I’m participating in AoC for.

View on HN

Correct.

Part 1: easy.

Apply part 1 solution to example, works.

Apply part 1 solution to real input, works.

Part 2: hard to very hard.

Apply part 2 solution to example, usually works.

Apply part 2 solution to real input, doesn't work due to time / memory constraints.

But having to look at the input to discover a pattern that isn't explained as part of the instructions always irks me.

I didn't enjoy the puzzles as much as previous years. My guess was they were structured in a way to make them non-trivial to solve with chat-GPT as the leaderboard is taken seriously by many.

Lots of people have speculated that this is true, but the creator of the puzzles (Topaz) has indicated that they did not concern themselves with Chat GPT or LLMs beyond specifying that if you do just use an LLM you should not attempt to claim top leaderboard spots which are for humans only.

In fact I didn't see people wrestling with LLMs trying to get them to solve early days and I can't believe it's impossible. My expectation is that the fad has passed, the sort of people who tried Chat GPT in 2022 because it was hot have moved on, the sort of people who resorted to it because they don't like AoC just didn't do AoC in 2023. If you enjoy these puzzles it makes sense for you to solve them, not to ask a machine to do it.

That's what many people who resorted to Z3 found frustrating. A Z3 solution to Day 24 is arguably "correct" and it certainly "works" but it's not very satisfying in terms of feeling like you achieved anything. This is why I wrote a solution which didn't do that even though it was days slower to write.

As a small counterpoint early on I do remember people trying and failing to use LLMs which prompted some of the speculation about the competition being made LLM-proof. As early as I think day 2 there was a big discussion about how no matter what people did the SotA LLMs could not give a correct solution for part 2. After that at least on the subreddit discussions involving LLMs were downvoted intentionally and so while presumably people were doing it there wasn't any discussion on Reddit at least.

I kind of agree with your last point. I try to do all the problems without non-built in python packages myself for exactly the reason you describe. Just feels more satisfying to me.

Interesting about the LLMs. I admit I don't read very much early AoC Reddit because I am most likely to be reading if I have questions after a few hours (like, wait, are these huge numbers prime or am I bad at arithmetic? Does my approach work and I screwed up, or am I an idiot and I'm on the wrong track entirely?) and in the first ten days or so I often don't have any questions.

Would you rather they re-skin the same puzzles people have memorized?

I’d rather they either give the assumptions in the puzzle description or at least in the examples. There were a few puzzles which I didn’t enjoy due to personal reasons but those I don’t have anything against, so long as the problem doesn’t boil down to „figure it out based on the real input since the example input doesn’t reflect it”

Reply on news.ycombinator.com