Hacker News Reader

Comment by Art9681 | Hacker News Reader

Seems like they need to adapt the level of difficulty to the new world of developer tooling. An AI may be able to write a starter template and used wisely can move the project forward, but it still requires knowledge and expertize to build a system of systems even with the AI.

This is an opportunity to change our expectations of developer potential. Up the stakes and innovation will soon follow.

Edit: This is also an oportunity for someone to swoop in and make the AI version of this competition with exciting and complex challenges, and make Advent of Code irrelevant by next year.

View on HN

>Seems like they need to adapt the level of difficulty

"They" is one guy - Eric Wastl. He has a very specific style and hey may not want to change the entire style of the challenge just because some dorks want to cheat. I hope that he doesn't change it -- AoC is very approachable, especially inside the first ~15 or so days. It's a charming set of puzzles that people look forward to. I don't want it to become an elite competitive programming competition just to stay ahead of LLMs. I want to be able to partake with 1-2 hours a night after my kids go to bed.

With that said, I don't think LLMs will be helpful with some of the tougher problems. It's a hard thing to test, because I assume CoPilot would do well with previous year challenges due to the massive amount of public code available on the same problems. I don't think it'd do well on designing the IntCode VM or the modular arithmetic problem on first time seeing them without examples on GitHub. But perhaps Eric has been feeding some LLMs the unseen problems for this year and the LLMs are cracking them? I'm skeptical but perhaps they've crossed that rubicon.

> With that said, I don't think LLMs will be helpful with some of the tougher problems. It's a hard thing to test, because I assume CoPilot would do well with previous year challenges due to the massive amount of public code available on the same problems.

I'm not 100% sure on this, but I think copilot isn't trained on anything past January 2022 (the gpt-3.5 and 4 cutoff dates), so I think it still shouldn't do well with AoC questions from 2022

You mean September 2021, right?

I'm basing that off this community post and a few other forums that noted that GPT knows about events after September 2021. Point stands either way.

https://community.openai.com/t/what-is-the-actual-cutoff-dat...

It seems the cut-off has shifted from September 21 to January 22 recently.

This entirely misses the point of AoC. We already have system-of-systems challenges - they're called work and they're not as fun as the self-contained whimsical puzzles that are AoC.

There is no reason they can't be. I don't know about you, but the system of systems challenges I deal with at work are certainly not whimsical.

I wonder whether this year's AOC authors are going to run their problem descriptions through Chat GPT first and adjust them to trip Chat GPT up.

Reply on news.ycombinator.com