Any ideas? The only thing I can think of is time after you saw the problem, which would work with an honor system on a social/friendly leaderboard.
Here's an example – in part 2 of day 1 of this year (2021), there's no need to actually add up the 3 measurement range, since the middle 2 values will always cancel out. You simply need to loop through the array compare index i with i+3. Maybe there should be an extra point for people who figured this out?
Another problem is that the input set is always small enough to make a brute force solution the most favorable one. So what they are really judging is the speed at which someone can read/parse the input file and write some loops.
Maybe one way it could do the performance testing is: 24 hours after the puzzle is announced, a "speed test input" becomes available. The speed test input would be substantially larger than normal inputs. Because people have had time to write the code, all they have to do is dump the input into the code and put the output into the box the fastest. A notable bottleneck is that every problem does have to be solved by the server ahead of time, and if it starts taking ten minutes to solve each instance then that's expensive unless you use very small pools of inputs.
That's only true for, well, ok maybe 1/2-2/3rds of the days. The rest require something more complicated, either with more complicated algorithms or more clever algorithms to get performance to a reasonable level. See the other discussion here about 2020 Day 13. The naive solution worked on the samples, but would not have worked on the actual input.
Obviously for the folks who use <ObscureLanguageX> this wouldn't work but it would be a nice collaboration otherwise.