Now, "running" may also mean different things. For example, you may want to do things s.a. performance diagnostics (in order to understand if your code uses resources efficiently), and then you'd need stuff like NVML, DCGM and co. Consumer-grade hardware might not be supported, or might not in principle support diagnostics collection / instrumentation. Or, if you have multiple GPU-dependent workloads that cannot saturate your resources -- you might think of MIG as being a way to address that... and, again, consumer-grade GPUs won't help you here...
I'm not saying you shouldn't try self-hosting. I'm actually all for it. But, you also need to be mindful of the pros and cons. NVidia must have some reason to pitch the datacenter family of GPUs to, well, datacenters. They aren't just blowing up prices.
Also, to give some sense of comparison:
Titan X ~= 3.5K CUDA cores.
V100 ~= 5K CUDA cores.
A100 ~= 7K CUDA cores.
H100 ~= 18.5K CUDA cores.
It probably doesn't translate directly into H100 being six times as fast as Titan X, but, I hear that these GPU workloads might be lengthy...
For some of these, you could obviously pay with your time. Sometimes that time is very valuable, and sometimes you have a lot to spare.
Also, the amount of VRAM (3x)... Sometimes having too little of it means having to re-write the program, or it could dramatically impact the speed. Similarly for bandwidth (8x). But, again, if the kind of workload you have isn't constrained by either, then you could probably win by running on more smaller / older GPUs.