3K, 60fps, 130ms: achieving it with Rust
blog.tonari.no
blog.tonari.no
Latency happens throughout the whole stack; Unfortunately much would need to be fixed outside this project to achieve any further significant improvement.
Operating System, firmware, blackbox hardware are some other non-negligible sources of latency. Everything adds up.
But 130ms, while much better than 500ms, is still bad. And I pointed outside the project as likely culprits.
It's a stretch to claim my post "slams other people's work".
I was reacting to the word "pathetic" in your original version. In the absence of disambiguating information about your intent [1], it sounded mean. If you had posted originally what you've got there now, it wouldn't have been a problem.
[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Yes, I edited it to make more positive, based on the reaction. It apparently was seen as as attacking the project, when that was far from my intent.
>I was reacting to your use of the word "pathetic".
I noticed this, and changed as soon as I could. I do believe 130ms really is pathetic and shows there's much work to be done (probably outside of the project, which I thought was clear in my original post but as per its reception, it definitely wasn't, so I expanded that part to put more weight on that), but do absolutely appreciate and welcome pretty much any work on reducing latency (endemic throughout software and hardware; everybody thinks 1ms here and there isn't important and it all adds up).
The 130ms is "the time from light hitting the camera to when it appears on-screen on the other side". The article doesn't really specify where the latency happens, but with just the encoding/decoding of video alone I think you'll very quickly get latencies in roughly the ~100ms ballpark no matter what.
Adding latency is always bad. You're seldom in control of the full stack. Latency will come from hardware, firmware, your OS, the network, everywhere. The responsibilities are split. But if even a single node of the chain doesn't take latency seriously, it screws up the whole, because every link of the chain adds latency.
That said, 130ms is a lot for a single link of the chain other than the network (physical distance will of course incur latency).
seL4[0] will guarantee (with formal proof) a low maximum latency despite multitasking.
Negligible, compared to that of the network.
>What improvements in "the stack" would be needed?
It would have to take latency seriously, thoroughly.
>The article doesn't really specify where the latency happens,
It really happens everywhere. The rust code likely accounts for very little of the overall latency anymore, but the overall latency is still bad.
I would pick seL4 because of their WCET proof.
It depends on the implementation. Most protected RTOSs are vastly inefficient but it doesn't have to be so.
For an example, many multitasking RTOSs do reserve CPU time for the "important" tasks, whether they use it or not. That's the industry standard for "mixed criticality", where less and more critical tasks share the same CPU.
But there's better ways to do this. Refer to the seL4 whitepaper[0].
>desktop OS
I'd rather not WaitForever™ for the computer to react to my input due to unbounded latency on my Desktop OS. We might be used to trash responsiveness, but interacting with a computer does not have to be this bad. We should stop pretending non-RTOS systems are okay to build a desktop around.
My point isn't to dismiss their work (any work on reducing latency is very welcome), but rather to highlight that 130ms is still bad, and that the rest of the stack (OS, firmware, blackbox hardware) is definitely a problem.
As I understand it, they have little in terms of outside dependencies, but still do run their software on a common OS.
Unlike seL4[0], Linux/OSX/Windows are unsuitable for real time.