Hacker News Reader

Comment by petesergeant | Hacker News Reader

I’ve been trying to design a puzzle for a game this year that humans can solve but LLMs can’t. I’ve come up with one, but it was hard work! It’s based around message cracking.

View on HN

There was one in a previous AoC that I think stumped a lot of AI at the time because it involved something that was similar to poker with the same terminology but different rules. The AI couldn't help but fall into a "this is poker" trap and make a solution that follows the standard rules.

Interesting! Maybe that’s the general way to approach these things

Was that 2023's Day 7 'Camel Cards' [1]?

[1] https://adventofcode.com/2023/day/7

I mean, wasn't pretty much the second half of all AoC exercises beyond LLM capabilities?

I remember there being multiple accounts trying to one-shot AoC and all ended on day 10 or so.

Isn't that easily solved by changing the terminology before giving it to the LLM?

Have a look at https://arcprize.org/

They have hundreds of challenges that humans can solve in under a minute which LLMs can not. Seems the general trend is figuring out the rules or patterns of the challenge when there are few examples and no instructions.

Ah, it also needs to be challenging for humans. It's a prize to win something. I just didn't want people to throw the question into Claude Code.

Perhaps coding exercises that require 2d or 3d thinking, or similar. This is where I have seen LLMs struggle a lot. There are probably other areas too.

For more examples of such problems check Jane street puzzles of the month

Those will almost certainly be too hard for the target audience

Just have to incorporate good judgement in some way.

How many <$letter>s are in the word <word with $letters>

The bigger LLMs have generally figured out this specific problem.

Reply on news.ycombinator.com