https://arcprize.org/leaderboard
$13.62 per task - so we need another 5-10 years for the price to run this to become reasonable?
But the real question is if they just fit the model to the benchmark.
$13.62 per task - so we need another 5-10 years for the price to run this to become reasonable?
But the real question is if they just fit the model to the benchmark.
At current rates, price per equivalent output is dropping at 99.9% over 5 years.
That's basically $0.01 in 5 years.
Does it really need to be that cheap to be worth it?
Keep in mind, $0.01 in 5 years is worth less than $0.01 today.
You could slow down the inference to make the task take longer, if $/sec matters.
But I don't think every developer is getting paid minimum wage either.
> Now that's a day's wage for an hour of work
For many developers in the US that can still be an hour's wage.