Detecting potential cheaters in Advent of Code Leaderboards | Hacker News Reader

Detecting potential cheaters in Advent of Code Leaderboards

adamfallon.com

18 pointsAJRF24 comments

Oh, it's about people who use AI, I thought they were going after the real cheaters: the people who use a prefab computer to solve the problems rather than building their own from scratch.

And describe tasks to the compiler, instead of assigning registers manually

This is an extremely bad faith reduction. It is a series of programming puzzles, not invent-the-universe problems, or paraphrase-problems-to-AI problems.

It's a joke

Yeah but I am fairly sure it is an AI generated joke - and I chose to only laugh at human created humour. I use an AI to check for me if the joke might be AI-funny and not real-funny. Then only if it is real funny might I humour it with a chuckle. Future comedians will thank me for my contribution in protecting genuine laughs worldwide.

Programming is telling the computer to do something, using the AI to generate code is just the next step up from a compiler.

It's also been a goal of CS for some time with some estabilished terminology. eg https://en.wikipedia.org/wiki/Program_synthesis & https://en.wikipedia.org/wiki/Natural-language_programming

My day 6 part 2 time was 46s, in a language fairly new to me and entirely manually (no automation at all). I just happened to have a solution that could quickly be changed to partition by 14 instead of 4

I’m not sure that just fast Part2 is that strong a signal. (I’m also not even sure that I think GPT is cheating any more than having pre-written frameworks, ready Dykstra, A*, and min-max solvers, automated input downloaders, scripted submission functions, etc. I don’t think any of those are cheating. GPT is more a grey area, I guess.)

It sounds like AOC is very formulaic if there’s a set of things to come prepared with.

Is it basically the same kinds of CS exam puzzles each year?

It’s going to have graph traversals, path optimization, input parsing, various flavors of map-reduce, and recursing every year.

In that sense, it has a definite pattern, but I wouldn’t say I experience it as formulaic.

Definitely agree on "recursing".

Every year I curse a lot, multiple times.

Also usually something with dynamic programming, something with primes, and something that you should/can solve with circular/doubly linked lists.

It is basically a CS class.

But the nice thing is that most puzzles have multiple solutions, it's always educational to read how other people did things!

I haven't been doing it this year, but looking at my answers from last year it looks like I have "cheated" day 24 part 2 and answered in 17 seconds (it was just a matter of adjusting a single range). There were also several "close calls" where it took me just slightly above one minute (day 2, 4 and 13).

> for a programmer looking up something like “network request python” things like ChatGPT or Github CoPilot become much easier to use, as they have IDE integration (Why didn’t Stack Overflow make IDE plugins for this stuff years ago?

No? isn't that just like Google "i'm feeling lucky" but without even bothering to look up who said what and why.

GPT strips things of context, and this makes it difficult to know how much to trust it, and value it in comparison to alternatives.

I mean if you don't mind quake3 code turning up in your work, and don't mind even knowing that's what it is and where it came from then i guess that level of don't give a crap matches what you are doing - seriously if it's just a hack i have nothing against this, but it doesn't feel right for any long living code.

Seemingly unpopular opinion: large language models are just a new higher level of abstraction we can use to describe what we want the computer to do.

Can I claim this guy is a fraud for using python instead of typing opcodes into a hex editor? He doesn’t even know what registers hold his data!

It’s yet another needless and error-prone abstraction that only works if it’s able to read human written code in the first place. I’d say it’s a pretty uninteresting, possibly dangerous, and ultimately distracting abstraction. My experience with ChatGPT was that it couldn’t ever tell me something that was actually correct, and I was asking it straightforward questions.

Large language models aren’t magic, they are tools. Most tools are potentially harmful in the hands of a novice and potentially powerful in the hands of an expert.

I didn't say they were magic. I said they're bad tools.

These tools seem like a needless middleman, for the most part. We'll see in ten years. Because ten years ago it was self-driving cars around the corner, and now we have the models saying full chat and knowledge providers are around the corner. I'll be very surprised if the edge cases, where stuff actually matters are ever addressed beyond demo uses or marketing/advertising applications.

The “fraud detection“ method used here was underwhelming. I was hoping for a statistical learning approach rather than a blunt heuristic. More justification for the heuristic would have been interesting.

> Using this API and a Python script, it is possible to mark any user who solves the second part less than a minute after the first as suspicious

Looks like for Day 25, this would have marked the entire leaderboard (top 100) as suspicious. (No, they didn't use ChatGPT.)

Part of the meta-game while solving part one is to predict what kind of parameterization part two might depend on and make a flexible solution, and the best solvers are also the best at doing that.

Maybe someone more knowledgeable can explain - is there a reason people would be concerned about cheaters in AoC? Is it just like for speedrunning, where the repercussions of cheating are largely about your perception in the community itself?

I guess what I’m trying to figure out is whether anyone’s going to try and get a job using their Top 10 Advent of Code medal.

There’s no real concern because AoC is just for fun. Cheating has always been easy but AoC isn’t a serious competition.

No one is going to get a job because of their AoC score, they’ll get a job because they have the knowledge and skill to complete the problems though.

I know a guy in the top 100 and a HFT recruiter reached out to him, mentioning his AoC score. Of course, his resume is already impressive as is

Why?

The leaderboard fills up in anywhere from 10 to 15 minutes. I bet it would take people at least as long to come up with a suitable prompt (just copying the text on the site doesn’t work), refine the response, and implement the solution.

Reply on news.ycombinator.com