Hacker News Reader

Comment by talldayo | Hacker News Reader

> Access serverless GPUs through a simple CLI to run your existing code on the cloud while being billed precisely for usage

Hmm... well I just watched you run nvidia-smi in a Mac terminal, which is a platform it's explicitly not supported on. My instant assumption is that your tool copies my code into a private server instance and communicates back and forth to run the commands.

Does this platform expose eGPU capabilities if my host machine supports it? Can I run raster workloads or network it with my own CUDA hardware? The actual way your tool and service connects isn't very clear to me and I assume other developers will be confused too.

View on HN

Great questions! To clarify the demo, we were ssh'd into a linux machine with no GPU.

Going into more details for how this works, we intercept communication between the CPU and the GPU so only GPU code and commands are sent across the network to a GPU that we are hosting. This way we are able to virtualize a remote GPU and make your computer think it's directly attached to that GPU.

We are not copying your CPU code and running it on our machines. The CPU code runs entirely on your instance (meaning no files need to be copied over or packages installed on the GPU machine). One of the benefits of this approach is that you can easily scale to a more / less powerful GPU without needing to setup a new server.

does this mean you have a customized/dummy kernel gpu driver?

will that cause system instability, say, if the network suddenly dropped?

We are not writing any kernel drivers, this runs entirely in userspace (this won't result in a crowdstrike level crash haha).

Given that, if the network suddenly dropped then only the process using the GPU would fail.

How do you do that exactly? Are you using eBPF or something else?

Also, for my ML workloads the most common bottleneck is GPU VRAM <-> RAM copies. Doesn't this dramatically increase latency? Or is it more like it increases latency on first data transfer, but as long as you dump everything into VRAM all at once at the beginning you're fine? I'd expect this wouldn't play super well with stuff like PyTorch data loaders, but would be curious to hear how you've faired when testing.

We intercept api calls and use our own implementation to forward them to a remote machine. No eBPF (which I believe need to run in the kernel).

As for latency, we've done a lot of work to minimize that as much as possible. You can see the performance we get running inference on BERT from huggingface here: https://youtu.be/qsOBFQZtsFM?t=64. It's still slower than local (mainly for training workloads) but not by as much as you'd expect. We're aiming to reach near parity in the next few months!

When you release a self-host version, what would be really neat would be to see it across HFT focused NICs that have huge TCP buffers...

https://www.arista.com/assets/data/pdf/HFT/HFTTradingNetwork...

Basically taking into account the large buffers and super-time-sensitive nature of HFT networking optimizations, I wonder if your TCP<-->GPU might benefit from both the HW and the learnings of NFT stylings?

Got it. eBPF module run as part of the kernel, but they're still user space programs.

I would would consider using a larger model for demonstrating inference performance as I have 7B models deployed on CPU at work, but GPU is still important training BERT size models.

Reply on news.ycombinator.com