If we try to benchmark GPQA-Diamond with DeepSeek-R1 in the suggested configuration of 0.6 temp and 32k max_tokens and say if every instance takes the maximum tokens it will require 6.4 M tokens. Which without batching on a single H100 at 80 tok/s will take 23 hrs to run. To run with 32k context length on a single H100 a 1.5B model will require ~15-20 GB VRAM so you cannot run 10s or 100s of queries in parallel.

MMLU-PRO is 12,000 instances. To avoid this we set a 600 seconds timeout for each instance to run.