Benchmarks in CI are hard, because you need them to run in the exact same environment to make any sort of conclusions. But CI's are often noisy, virtualised, dockerized, whateverized. There is not much benefit in that.
Not ideal but it's better than pure X vs Y.
It could still be a good, but very expensive, service, but its hard for me to imagine something that would be one-size-fits all service that would accurately predict real world experience.
It wouldn't be much harder than just making sure you're the only VM running on the hardware (disk & cpu) . Much like a "reserved instance", with real guarantees against side effects.
Obviously you can build test suites for each of these scenarios, but I think it would be expensive to run all of them. That's all I'm really saying, it's by no means a bad idea, I think it's a great idea, just going to require some upfront thought of what type of environment the software is going to run in.
So it's expensive b/c it consumes CPU and time on shared systems, and those cost money... so I think a service like this could potentially cost significantly more to operate than say travis-ci.