Hacker News Reader

Comment by radarsat1 | Hacker News Reader

I'm confused, if this operates at the CPU/GPU boundary doesn't it create a massive I/O bottleneck for any dataset that doesn't fit into VRAM? I'm probably misunderstanding how it works but if it intercepts GPU i/o then it must stream your entire dataset on every epoch to a remote machine, which sounds wasteful, probably I'm not getting this right.

View on HN

That understanding of the system is correct. To make it practical we've implemented a bunch of optimizations to minimize I/O cost. You can see how it performs on inference with BERT here: https://youtu.be/qsOBFQZtsFM?t=69.

The overheads are larger for training compared to inference, and we are implementing more optimizations to approach native performance.

Aah ok thanks, that was my basic misunderstanding, my mind just jumped straight to my current training needs but for inference it makes a lot of sense. Thanks for the clarification.

> to approach native performance.

The same way one "approaches the sun" when they take the stairs?

I guess there's non-negligible optimization potential, e.g. by doing hash-based caching. If the same data gets uploaded twice they can have the blob already sitting somewhere closer to the machine.

yes. my building has stairs and i find them useful because usually i don't need to go to the sun

Is DirectX support possible any time soon? This would be huge for Windows VMs on Linux...

You could use unofficial Windows drivers for virtio-gpu, that are specifically intended for VM use.

Indeed, it's just that it's not very feature-complete or stable yet.

Reply on news.ycombinator.com