Latency testing remote browsing: Why display streaming is hard
thume.ca
thume.ca
[0] https://twitter.com/trishume/status/1591995028842418176?t=Gz...
and even if "cloud" "gaming" becomes the new normal shoved down our throats anyway, there are enough great games for a few lifetimes already
This is merely premature. We'll eventually get GPUs at the edge to serve urban customers on a wide variety of devices.
I'm not a fan of subscription gaming, but companies will want to do this for their entire back catalogs.
> GPU mining in sharp decline, that seems to be unlikely to happen soon. the GPU market recovering.
Just in time for all the AI/ML startups!
publishers love the idea, yeah, but people don't. until it's outright forced upon us, it will not gain traction
>Just in time for all the AI/ML startups!
thankfully, not a great time for bullshit startups, and hopefully won't be for a while longer
I don't know what the future holds, but right now, things are looking up. for example, a 3060 is $350-400 and there's no shortage of cards.
This Thume guy gets a lot wrong.
I'm not going to spend eons taking it apart: his ridiculous breadboard thing does not measure something precisely or accurately enough to be useful, and whatever it did measure he misinterpreted anyway.
> Encoding the whole window means latency scales with window size.
> something called “damage regions” that apps provide to the OS compositor
> possible to diff tiles of a 4k screen image in under 2ms
> One disadvantage of the hardware accelerated video encode approach is that using minimal damage regions is tricky, because all you have is a vendor-provided API to encode new full frames into a video stream, and you can’t count on the vendor to optimize their encoder for large pixel-equal screen regions getting encoded very quickly.
This is just so far off base. Until he checked the "Enable H264 encoding" box, the stream was compressed on the CPU, to VP8. So it's not even hardware accelerated. It's just slow to compress on the CPU, period full stop, which of course scales with screen size much more than the dedicated compression hardware on their VMs.
The low performance has nothing to do with "minimal damage" regions, that provides no meaningful benefit to a hardware encoded stream. An H264/H265 P frame is much more optimal than a "minimal damage" region, it's like comparing spaceflight to a horse drawn carriage.
Large pixel equal screen regions are of course encoded extremely well in something like NvEnc and Intel QuickSync. What vendors is he even talking about?
> Does it constantly send 60fps compositor updates? No. A lot of apps make this mistake but Remotey doesn’t. If you mess it up it causes WindowServer to take an extra 10-20% CPU and extra power
This gets the biggest facepalm to me. This is the same crowd that thinks Unity games are slow because of "Gfx.WaitForPresentOnGfxThread."
Enough about Thume. The simple reason Mighty failed is that it MADE. NO. SENSE. Their latency could have been zero, and approximately zero people would have still used it. I'm not going to toot my own horn here, there are definitely streaming business that not only make sense, but make a HUGE amount of $$$ SENSE, but remote browser streaming isn't one of them.
With a small macroblock size a large enough vertical scroll might cause the encoder to send fresh pixel data for the blocks with a large enough vertical translation. A context aware encoder would not only not miss the translation but also know where to begin its search algorithm, improving the encode speed.
In another case a window coming into view, misaligned to a macroblock boundary, would need to send more data than a context aware encoder that simply sent the window pixels and the location to draw them.
Perhaps you can tell us what's wrong with his "ridiculous breadboard thing" rather than just stating that it's ridiculous?
About damage tracking and hardware encoding: The encoder must always diff the whole frame. There is no magic involved here. It probably does it intelligently, e.g. by hashing blocks and comparing the hashes rather than the memory regions. However, comparing the whole frame still means that you must read the entire frame from memory.
He is correct in saying that an encoder that is aware of damage regions would be more efficient. This is because it would require a lot less memory bandwidth.
He is also correct about 60 fps compositor updates. You probably just misinterpreted his meaning. What he's saying is that when there is nothing happening on screen, you shouldn't tell the compositor to keep updating the content because that just wastes energy on compositing something that hasn't changed.
However, this is tricky when you have a constant bandwidth h264 stream because you will have incremental updates for the same frame. A trick I've used to get past this is to stop encoding when the size of the encoded packages goes below a certain threshold, at which point the image isn't going to improve much.