github.com
Ah, disappointing; I was hoping to try this with a Steam Deck as an alternative to using Moonlight and streaming the entire game from the Windows machine.
I assume it cost quite some $$$ to produce this because you kinda have to cut nvidias binary drivers in half, which is going to require quite a lot of reverse engineering.
We've got some of the best devs working on this and we hope to impress!
In particular:
- Autoscaling Stable Diffusion Inference
- Traditional creative workflows (realtime GPU viewport in octane for example)
- Gaming from one GPU in your house to everywhere else
I get the training example for small models but can't imagine it scales that well with model size
The big value seems to be... share 1 GPU to many computers, so spend less on a cluster? Capacity fungibility is real value but hard to measure!
In any case, stuff like this is a good bet. GPU software will continue to increase in prevalence, and utilization will remain low. Solving for the compute market liquidity is important despite NVIDIA's best efforts.
I have to imagine a lot of ML infra today is built for Big Dedicated Deployments and not necessarily friendly with more serverless architectures.
That is to say, I'd guess a robust version of this has it's use cases - whether that value prop is in DX, autoscaling, architecture simplification... I'm not sure.
Many of them support the kserve API standard[0] that supports everything from model loading/unloading to (of course) inference requests across models, versions, frameworks, etc.
So in the case of Triton[1] you can have any number of different TensorFlow/torch/tensorrt/onnx/etc models, versions, and variants. You can have one or more Triton instances running on hardware with access to local GPUs (for this example). Then you can put standard REST and or grpc load balancers (or whatever you want) in front of them, hit them via another API, whatever.
Now all your applications need to do to perform inference is do an HTTP POST (or use a client[2]) to a Triton endpoint URL for model input, Triton runs it on a GPU (or CPU if you want), and you get back whatever the model output is. So now everything else in your architecture other than Triton doesn't even know what a GPU or ML is.
It also makes deploying new models, versions, whatever much simpler - you POST them to Triton (or it loads them from S3, local disk, whatever) and they're instantly available everywhere.
Not a sales pitch for Triton but it (like some others) can also do things like dynamic batching with QoS parameters, automated model profiling and performance optimization[3], really granular control over resources, response caching, python middleware for application/biz logic, accelerated media processing with Nvidia DALI, memory management and control, all kinds of stuff.
[0] - https://github.com/kserve/kserve
[1] - https://github.com/triton-inference-server/server
[2] - https://github.com/triton-inference-server/client
[3] - https://github.com/triton-inference-server/model_analyzer
Even just the time it takes speed of light between NY and LA (410^6m/310^8m/s=1/75s) is roughly how long a 60 fps frame is (1/60s). Add OS serializing the frame from the GPU onto the network card, network switching of those packets, and you're starting to really feel that latency.
1) Take off the glass
2) Use a drill, make the connect wider
3) now bundle 9 glass and put in hole
4) now bandwidth is more wide
Fasterness!
Unless there is a need to evict all gpu memory on every frame, I think it is feasible to game on GPUs that live across a very fast LAN.
Definitely of interest to us, even w/ latency limits, both for ai dev & investigations and occasional full runs
I do have to wonder about the non-oss licensing, as that's part of why we didn't spend much time on bitfusion...
Also for most real GPU applications, you need to get the data in and out. I don't think splitting compute across a (insert any non-Infiniband-link) solves this