The course had a machine solely for benchmarking. It would only compile and benchmark one program at a time and the students had to queue up.

It worked well for the course and the results were very consistent between runs, but that kind of setup doesn't scale well.

There may be a (slow) option though, and that would be benchmarking the code using an simulated processor. The runtime would be measured relative to the simulated processor as opposed to the computer hosting the simulator.