Advent of Code 2023's new AI/LLM Policy
adventofcode.com
adventofcode.com
I understand the appeal of using the best and fastest means when there are real-world stakes — taking a motorcycle is better than running for many transport scenarios. But the whole point of this is to have fun! Is it fun to hit <tab> over and over?
In the case of AoC, the only objective incentive I can think about is impressing a recruiter. Maybe it's enough if you're motivated enough.
I play chess and some players end up thinking that cheating allows them to reach their real level and that it's just bad luck or conspiracy that prevent them to get to that level without resorting to cheating. Talk about cognitive dissonance.
Designing such a robot would be an achievement in its own right, but not really in the spirit of the competition.
If you just rented some time on someone else's robot, its just plain cheating.
I would love to see teams compete, in a separate venue/ranking, based on their own model's ability to write code with minimal prompting.
It would be kinda cool to see what can be done in different 'classes' of model. How far can you get with a local model running on an 8gb card, or just using an M2 mac. How far can you get with a custom tuned chatgpt vs a programming specialized model. Etc...
Yes, an AIdvent of Code would be interesting...
There's two views: (1) programming as a craft and an art, a human creation, and (2) programming as a hurdle to overcome to accomplish more important things. People with both views might wish to practice.
Last year, AI was only able to solve the first 3 or 4 problems, so it's really not like using AI takes all human ingenuity out of the equation. What it does is enable you to autocomplete sub-problems that are already solved problems so you can focus on the good stuff. I'd personally prefer if there were 2 leaderboards - one with AI, and one without. Motorcycle races are fun too :)
That's not to say that you can't still do a foot race between A and B. It's just now you have to constrain it, it has become something different.
Weirdly enough, basically this happens all the time at races [0]. Not a motorcycle, but a runner in my local competitive scene used a bicycle to fake her strava data for a HM in 2017 [1].
I don't get it either.
[0] https://www.wired.com/story/marathon-investigation-cheaters-...
[1] https://washingtonpost.com/news/morning-mix/wp/2017/02/23/ho...
It's also likely that a lot of AI users will be skilled developers too. They already know they can solve the problems, they also think they deserve the leaderboard rankings, so using AI is ok.
This reminds me of the Project Euler position on people posting their solutions publicly. The assumption is others would seek them out, get credit they don’t deserve and not learn as much.
I understand trying to prevent skewing the results, but personally it doesn’t affect me since I’m not doing it to compete in the website, I’m doing it for self edification.
I believe this actually helps me compete in the thing that really matters, getting a job, bringing high quality products to market and generally solving real problems people have.
Are LLMs starting to solve some of that? Sure. So then I want to compete for the jobs that still can’t be solved by them and involve more complicated and interesting solutions.
If people want to handicap their problem solving abilities or compete for jobs supervising boilerplate generation, all the better for me. I’ll take a hopefully more rewarding career over making the leaderboard on AoC and PE. Though I do feel sorry for their maintainers.
> Is it fun to hit <tab> over and over?
It’s probably a mixture of entertainment (I’ve seen countless apps where it looks like people just tap the screen as fast as possible without even really watching what’s going on) and imagining they’re learning something by watching answers appear (although IMO the lack of engagement leads to lack of retention; sort of like Socrates’ position on reading vs discussion).
Almost. But the crucial difference is — imagine an alternative universe, where humanity didn't have horses or any other animals or devices, even rickshas or wheels, to help moving around the world, and suddenly the first primitive cars were being invented just a couple of years ago.
Wouldn't you feel enthusiastic to use those new devices and test them everywhere?
Bringing a motorcycle to a 5K is obviously cheating. The motorcycle is a factor of 10 faster than its closest competitor. That is not where LLM/AI is currently at. The code generated by AI is not 10 times better than human code. It is just human code cleverly regurgitated.
Comparing AI/LLM to the motorcycle is giving it too much credit. AI is more like a 500CC motorcycle vs a 300CC motorcycle.
Sure they are pretty close, and the human might win once in a while, but the AI has just enough of an advantage that is guaranteed to burn the human out and win 95+% of the time.
I use copilot a lot and use chatgpt a lot to help with coding problems and doing things like writing short bash scripts for me, so I feel like I have a good grasp of what it's capable of doing, and the amount of work you'd have to do to break the problem into small enough chunks that it can understand and solve it is similar, if not more, than the amount of work you'd have to do to just solve it yourself. At least in its current state.
I think it defeats the purpose of the AoC to use AI, but I can imagine for some people it's a different game to see how they can use a screw (gaming term) to circumvent the system.
On the other hand, there may be some value to learning how to apply AI tools as a solver. If test cases are available for a given problem space, and AI can be applied to build code to meet those "requirements", then it is probably a skill we should be learning.
While I don't believe people are going to play fair here, it would be nice if AoC submissions had an [AI-assist] checkbox. Then there could be two leaderboards, and the AI folks could still have some good competition with each other.
Ultimately, the AoC leaderboards are unfair as they favor people in timezones that fit the release of each "day". So being on the leaderboard isn't really something that many people can do, even if they are brilliant problem solvers.
I'd say that whatever "better at programming" is quickly evolving. AI is a tool like high level languages, IDEs, autocomplete or all the different types of code analyzers. How is ignoring a powerful tool like AI going to get you better at programming?
Programmers who think they are better programmers without AI may quickly be replaced with "lesser" programmers who get things done much quicker.
I think there's a both-and answer for the contest. Maybe have one competition where the unenforceable spirit of it is, don't use LLMs for help, and another one where the challenges are just made... quite harder... so that even people who use the LLMs still need to marshal great ingenuity to use them better than others (e.g. the #1 spot is someone who used RAG and chain-of-thought better than the #2 spot, and also had better intuition of what to trust vs challenge from the LLM outputs).
By that token, using AI to solve AoC really seems like one would be doing themselves a disservice. You might tell yourself you fully understand what the AI spits out, and maybe you do, but you won't build the same sort of comprehension that you would from solving it yourself. Nor will you retain it for as long.
> Software engineers who think they are better without AI may quickly be replaced with "lesser" software engineers who get things done much quicker.
Using AI to program is just ctrl-c ctrl-v software engineering, very clearly not the point of programming competitions. High level tools as you mention don't make you a better programmer, they make you a better software engineer.
I think the discussion you're raising is a good one it's just not relevant to a programming competition.
I've been thinking about this for a full year now, and I'm ready to present my opinion: I think AI assistance makes it easier, not harder to get better at programming.
Sure, if you're lazy you won't learn anything because you'll get the AI to do all the work for you. But if you're that lazy I imagine you wouldn't have even tried to learn programming without access to AI assistance, so you're still doing better here!
If you're curious and dedicated to self-education, LLM assistance is a gift that never stops giving. You have to learn HOW to learn with it first, but once you've figured that out you can really fly with it.
The ”illusion of competence” effect can be really strong, and it sneaks up on you without you noticing it.
Highly recommend it. Very much.
if i'm trying to practice handwriting -- why would i use a type writer, or word processor?
i am not sure how using a machine to do the work you're trying to practice makes you better at that practice, since it's doing the.. mechanics.. for you.
as another commenter pointed out i feel there is a distinction between programmer vs. software engineer just as there is a distinction between writing and say, novelist.
i understand the notion of practicing, and getting better at an evolved workflow for producing results -- but we're practicing foundational skills, not trying to reduce everything simply to achieving results by any means necessary.
i feel that producing results without understanding is a short-sighted investment, and not exactly the point of AoC imo.
Is the difference between telling someone to do something versus asking how to do it.
The human brain/body requires resistance/challenges, otherwise it will always choose the lazy road and won’t improve.
Programmers that write "slow C" instead of "fast assembly" were once regarded as lazy and incompetent, same for programmers that write "slow Java instead of fast C" or "slow Python instead of fast Java."
> (If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)
This feels very similar.
For day-to-day use, as long as you know what the code is doing when you run it, its fine. But to get to the point where you understand what the code is doing, you have to have written code manually.
I think that'd exclude most Python, Ruby, Java, JS, Haskell,... developers. Heck, it'd even exclude a bunch of C developers who don't appreciate the fact they're not working on a PDP-11.
Yes, using machines to give an answer, without providing any explanations of the process is not learning. One cannot generally give a young kid a calculator and expect them to learn how adder in the calculator's ALU works.
However, using machines to get an answer that you know how to get to is beneficial. No learning is involved in either process, and it saves time. So if you give a kid, who already knows well how to perform addition, understands the principles, and knows the properties, a calculator to quickly get through their homework, it merely saves their time. You know, just like most people don't exactly like to write boilerplate code that they can type in their sleep.
And using machines that can provide explanations how they got to the answer then reading it is learning. And learning by example could be powerful in certain situations - e.g. when one already knows some stuff, but needs some hints how to apply it to a different problem.
Surely, just asking the machine to solve a puzzle is unlikely to get one anywhere. Asking the machine to guide towards a solution might be very beneficial (if it manages to get things right, of course).
That's just abstraction ? We're constantly dealing with legacy abstractions built 30-40+ years ago when we had fundamentally different constraints from today. Leaky abstractions are a fact of life and there's only so much you can reliably know about the stack.
My problem with learning with LLMs is how bad they are at the moment - people are betting big that they will eventually get better - but so far I'm only seeing scaling walls and negative cost/benefit footed by big tech $$$.
(This isn't theoretical - I've done Advent of Code a few times, but I don't get to code too much these days. I am curious about its abilities and having a fun way to approach it might be a reason to do Advent of Code.)
That said, I'm not trying to push to the top of the leaderboard.
Seems sensible.
Like, I'm sure there are some professional CNC operators who are also very, very good at doing things by hand. But they didn't get good at doing things by hand by using a CNC machine to do everything that can be done on a CNC machine. And, based on what I observed growing up in a down-and-out manufacturing town, the people who decided that CNC is the way of the future and went all in on just being very, very good CNC operators are the ones who got hit hardest by the growth of industrial automation in the 1990s and 2000s.
I have similar concerns about Copilot. In the short term, yes, it lets you get relatively straightforward tasks done more quickly. But I suspect that people who make heavy use of Copilot risk plateauing at a lower skill level and ultimately being the people whose jobs are most vulnerable to the growth of technologies like Copilot.
This is an opportunity to change our expectations of developer potential. Up the stakes and innovation will soon follow.
Edit: This is also an oportunity for someone to swoop in and make the AI version of this competition with exciting and complex challenges, and make Advent of Code irrelevant by next year.
"They" is one guy - Eric Wastl. He has a very specific style and hey may not want to change the entire style of the challenge just because some dorks want to cheat. I hope that he doesn't change it -- AoC is very approachable, especially inside the first ~15 or so days. It's a charming set of puzzles that people look forward to. I don't want it to become an elite competitive programming competition just to stay ahead of LLMs. I want to be able to partake with 1-2 hours a night after my kids go to bed.
With that said, I don't think LLMs will be helpful with some of the tougher problems. It's a hard thing to test, because I assume CoPilot would do well with previous year challenges due to the massive amount of public code available on the same problems. I don't think it'd do well on designing the IntCode VM or the modular arithmetic problem on first time seeing them without examples on GitHub. But perhaps Eric has been feeding some LLMs the unseen problems for this year and the LLMs are cracking them? I'm skeptical but perhaps they've crossed that rubicon.
I'm not 100% sure on this, but I think copilot isn't trained on anything past January 2022 (the gpt-3.5 and 4 cutoff dates), so I think it still shouldn't do well with AoC questions from 2022
https://community.openai.com/t/what-is-the-actual-cutoff-dat...
1. The policy makes sense to me.
2. It would be fun to have an AI leaderboard too. It was a decent amount of engineering work to win last year — I had 25 LLMs going in parallel, then running their solutions against the sample problem in attempt to validate the solutions. So there is still a lot of space for innovation.
For those interested, I published a version of the code at https://github.com/max-sixty/aoc-gpt.
I used ChatGPT (3.5 - 4 wasn't out yet) to help with Advent of Code last year and found it extremely useful - but I was careful not to attempt to make the leaderboard because it felt unfair to me: https://simonwillison.net/2022/Dec/5/rust-chatgpt-copilot/
I gave up after day 15 because it got too time consuming, with or without ChatGPT help.
> Can I use AI to get on the global leaderboard? Please don't use AI / LLMs (like GPT) to automatically solve a day's puzzles until that day's global leaderboards are full. By "automatically", I mean using AI to do most or all of the puzzle solving, like handing the puzzle text directly to an LLM. The leaderboards are for human competitors; if you want to compare the speed of your AI solver with others, please do so elsewhere. (If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)
Being in the UK I've never bothered attempting to target the global leader-board as 5am is just too early.
According to my research I probably could scrape in here or there (looking at the time between the timestamp on my input file - which I download by running a script manually - and my solution times) but then I assume I'd be one of thousands who could also do the same but are hampered by timezones.
I think the "Please don't" wording of the policy is pretty much all that Eric can do.
People will always try and game the system, but I doubt they can consistently game it for the number of days required to appear on the final leaderboard. I think the problems will just be too big/varied for LLMs to consistently solve.
I haven't even bothered with the private leaderboards as friends have varied commitments that still adds a time advantage to others.
AoC has always been about competing against myself
This is also shown in the Codex paper, where they trained an LLM to write code and then watched it solve a number of code problems they handwrote originally to make sure the problems could not have been in the training data.
Try it out yourself, make up some little math word problems and ask chatGPT or something.
Of course, advent of code will be much more challenging problems, but to get help with some subcomponents of the problem a motivated participant would likely try to use the most recent, powerful, and advanced models which outperform the results from papers written a few years ago, and outperform the free chatGPT.
I have, and chatGPT simply sucks at logical reasoning in the way that is required for mathematics. It's extremely easy to make it trip up by just formulating questions in a slightly nonstandard way (so that it doesn't exactly match what it has been trained on), e.g.
> User: prove that there are only finitely many primes
> ChatGPT: Actually, it is not possible to prove that there are only finitely many prime numbers because this statement is false. There are infinitely many prime numbers, and this has been proven by the ancient Greek mathematician Euclid over 2,000 years ago. Euclid's proof is based on the principle that if you assume there are only finitely many prime numbers, you can always find a new prime by multiplying all known primes together and adding 1. This new number will either be prime itself or have a prime factor not present in the original list of primes, which contradicts the assumption of finiteness.
> User: prove that there exists a finite list of integers that contains all integers which are only divisible by itself and 1
> ChatGPT: You're referring to prime numbers. The statement is essentially the definition of prime numbers. A prime number is a positive integer greater than 1 that is only divisible by 1 and itself. There is no need to prove this statement because it defines what prime numbers are. While there are infinitely many prime numbers, they can be listed, but this list will be finite in the sense that there are infinitely many prime numbers, but they can all be enumerated within an infinite sequence. Some of the first prime numbers are 2, 3, 5, 7, 11, 13, and so on.
ChatGPT gets the first question right because something like that is undoubtedly in its training corpus, but it gets utterly confused by the second question (which is the same statement), to the point that its answer belongs to the "not even wrong" category.
I don't know where this myth comes from that LLMs are magically good at maths. They're not.
Secondly, quality is highly variant and there are traps the context window can fall into which causes especially bad results. Jeremy Howard has a great video (https://www.youtube.com/watch?v=jkrNMKz9pWU, starting at 18:05 the 'limitations and capabilities' section is only 13 minutes long) talking about how quality depends on: how you frame your prompts, model power (4 does a lot of stuff that 3.5 can't), and whether you're in a kind of "context trap" of repeated mistakes.
Of course, some people like to point out that if it's so "finicky" and variant, it is "dumb." Sure, if you like. I'm not interested in whatever definitions you're using those things, the objective and observable point is that given well-known prompting practices, LLMs can do something functionally equivalent to reasoning about novel problems, and more powerful ones can reason about more powerful and difficult things.
I re-phrased your prompt (instead of "prove a false thing" I made it like "decide whether this thing can exist, and prove your answer"). And added a little well-known boilerplate prompt sugar. It seems to have done a better job.
https://chat.openai.com/share/53214f0c-17f7-4a3d-95be-8fd676...
But even given "careful prompt engineering" it's still extremely easy to make ChatGPT trip up (I don't have access to GPT-4, so let's talk about ChatGPT for now):
> User: User I am a student of machine learning and I have a graduate-level of understanding of certain subfields of machine learning and mathematics. I would like to ask you a tricky question about math. Please think carefully and step-by-step. Because, as a language model, each token you generate is a way to perform more computation, please write out your understanding of the problem and your steps of thinking about it. The challenge is as follows: decide whether or not there exists an algorithm that can decide for any statement in propositional logic whether it's true or not
> ChatGPT: The problem you're describing is known as the decision problem for propositional logic, or the "satisfiability problem" (SAT). It asks whether there exists an algorithm that can determine, for any given statement in propositional logic, whether it's true or not. This problem is fundamental in computer science and mathematical logic.
> I will walk you through the key concepts and steps in thinking about this problem:
> Propositional Logic: In propositional logic, statements are composed of variables and logical connectives (AND, OR, NOT, etc.), and the goal is to determine whether a given combination of truth values for these variables makes the statement true (satisfiable) or false (unsatisfiable).
> Formal Statement of SAT: The SAT problem can be formally stated as follows: Given a propositional formula (a statement in propositional logic), can we design an algorithm that correctly decides whether there exists an assignment of truth values to the variables that makes the formula true?
> Assume There Exists Such an Algorithm: Let's assume, for the sake of argument, that there exists an algorithm SAT that can decide the satisfiability problem for any propositional formula.
> Constructing a Contradiction: Now, we'll use a technique similar to the one used in the proof of the Halting Problem to create a contradiction. We define a new formula, Q, that represents a situation where SAT contradicts itself. We construct Q as follows:
> Q(P):
> if SAT(P) == True:
> NOT(P) # Negation of the input formula P
> else:
> P
> In other words, if SAT says that P is satisfiable, Q takes the negation of P, making it unsatisfiable, and if SAT says that P is unsatisfiable, Q takes P itself.
> Consider Q(Q): Now, let's consider what happens when we apply Q to itself, i.e., Q(Q).
> If SAT(Q) == True, then Q(Q) should be unsatisfiable because it takes the negation of itself.
> If SAT(Q) == False, then Q(Q) should be satisfiable because it takes itself.
> Contradiction: We've reached a contradiction. If SAT(Q) == True, then Q(Q) should be unsatisfiable, but our definition of Q says it should be satisfiable. If SAT(Q) == False, then Q(Q) should be satisfiable, but our definition of Q says it should be unsatisfiable. This contradiction arises because SAT cannot consistently determine the satisfiability of Q(Q).
> Conclusion: As in the Halting Problem, we've demonstrated that there are cases where the algorithm SAT fails to give a correct answer. Therefore, we can conclude that SAT (or any similar algorithm) cannot exist to solve the satisfiability problem for all cases.
> This proof shows that there is no general algorithm that can decide the satisfiability of any propositional formula. The SAT problem is known to be NP-complete, which means it's one of the most challenging problems in computational complexity theory. While efficient algorithms exist for many instances, there's no guarantee of a general solution for all possible cases. This has significant implications for problems in complexity theory and computer science, including issues related to algorithmic efficiency and the limits of computation.
This is absolute garbage. Even worse than garbage, it adds the additional insult of being plausibly-sounding garbage to people who've had maybe a cursory introduction to theoretical CS. And notice that it's not even that hard to come up with such an example: a question like this could plausibly be on a problem set for an intro theoretical CS course (if it's not too easy for even that).
My main point is that this betrays an absolute incapability of ChatGPT to perform basic deductive reasoning reliably. It clearly knows that the referenced statement is about SAT, and, if you prompt it directly, it also knows that SAT is decidable, but it somehow isn't able to connect these two facts.
BTW my objection was related to mathematics, not coding. It's possible that, in the average case, LLMs perform much better at coding, since the level of rigour required in many cases is less than in mathematics. But when it comes to pure deductive reasoning, I wouldn't trust LLMs. Or at least not ChatGPT.
Using LLMs to "win" at Advent of Code is - for me - a bit like getting an app to solve a Sudoku for you. You've got to end state pretty quick, and building the app to solve Sudokus is fun (trust me), but when you are trying to have fun and stretch your brain by solving a puzzle, how has that helped?
I think Advent of Code for me is about having fun trying to solve problems in my head and with my favourite IDE and programming language while having my first cup of tea of the day. I wouldn't want to use an LLM. I don't care about the global leaderboards.
I definitely see though how using one could be fun. And building code that can drive an LLM to solve the problem sounds like a ton of fun. So it would be nice to allow those who want to do that to flag it as such and to have their own leaderboards, or for it to show in leaderboards and allow them to be filtered in/out.
And like I say, at some point it will feel as weird as saying "no compilers".
Most people are not realistically shooting for a spot on the leaderboard.
It adds novelty value to know that some geniuses are out there who can solve a problem in 20 minutes which takes me over 4 hours, but the main value is my private leaderboard, and the enjoyment of solving the puzzles.
It's pretty unlikely for AOC to penalize people who use LLMs in any way (they have no way of detecting such use). This means there's going to be two groups of competitors, the honest ones and the ruthless ones. The ruthless group will probably be much smaller, but, considering how popular AoC is, probably big enough to completely overwhelm the honest group.
To be clear, the "ruthless" group won't be clueless people using Chat GPT for absolutely everything, but people who could very well solve the tasks on their own and figure out a way to leverage LLMs as efficiently as possible.
A rule permitting LLMs would give everyone an equal footing.
We can also learn a thing or two about prompt engineering.
I might try a wholly separate challenge this year: seeing if I can talk an AI into solving each of the puzzles. I wonder how far I could get using only ChatGPT's output without any hand-coding.
I'd use LLMs to help unblock me in the same way I use Google, since lately I've been finding it more difficult to find easy answers to "how do I do X with Y library" with Google. I've been finding the first page of Google showing SEO pages for programming questions rather than Stack Overflow or official documentation.
I wouldn't just copy-paste the problem into ChatGPT and then use prompts to solve it. I want to have fun solving the puzzle, rather than asking ChatGPT what the answer to the puzzle would be.
If your intent is to make it a pure human-keyboard-code challenge, then I would have thought this would be a great opportunity to make the underlying questions harder, or to find the edge cases (in terms of input or question format) where LLMs are no help.
In the same way, a human competition in advent of code is an interesting challenge--and wouldn't be, if people cheat with AI solvers.
The point of AoC was always to practice something new, learn a couple of things, challenge yourself a bit, share solutions and ideas with people etc. I don't see how that's changed by the existence of LLMs or by this policy.
Isn't every coding competition pointless busywork? The organisers already know the solution, and even if they don't I can probably find a dozen solutions on github/reddit/wherever.
The point is you can enjoy it even though it's pointless busywork - much like you can enjoy doing a crossword, or running a marathon, or lifting a heavy weight in the gym.
If you can't solve the problem in a few seconds/minutes you will be able to just look up or copy/paste the answer for every AoC day from the associated solutions thread.
So in other words if you view it like that there was already a better "tool" to very easily "solve" any AoC problem: internet search.
If all you care about is getting a number to put into the website to get a checkmark that is.
Even then, it seems like something being beatable by a computer doesn't make it pointless to do as an exercise. We still teach children to add and multiply, even though computers can do these things billions of times faster than a human can and have been able to do so for decades. We believe that being able to do these things in ones own head is valuable even if it isn't the fastest way of doing them. Besides, there are still many people who practice and enjoy chess even after computers beat humans at it, and in fact many people integrate computer chess programs into their practice and analysis of chess.
The advent of code problems are already "busy work", in the sense of not making much sense, and having numerous complexities thrown into them for the sake of it. People have still enjoyed them for the competitive aspect, or for the fun of solving a puzzle. Those aspects don't change when you introduce an AI that can solve them faster.
Nothing to worry about if new programmers are learning from AI that writes, on average, code much worse than anyone I know.
There was always the opion of not doing your work yourself, and instead paying a few bucks on some site to have someone else do it. Same crowd, it just got cheaper.