You can check Exo out:

https://github.com/exo-explore/exo

It's a project designed to run a large model in a distributed manner. My need for GPU is to run my own machine learning research pet project (mostly evolutionary neuron network models for now), and it's a bit different from inferencing needs. Training is yet another different story.

But yeah, I agreed. I think machine learning should be distributed more in the future.