Ask HN: ML models on live video stream for production

1 pointyshanks3 comments

I was trying to run a deep learning model on a live video stream in the server (and then send that video back to client). I tried using aiortc for this but doesn't seem scalable. Does anyone know other ways?

View on HN

You could try running the model in the browser with Tensorflow.js. I recently tried this for something like that and found the overall workflow much easier. TFJS tooling is pretty good.

Before I had tried using custom setups with Janus in the loop and it was slow going. AioRTC flat out never worked for me.

I am trying to run a segmentation model and tfjs is too slow for that. How do people do it on the server side? Something like zoom does with virtual background.

I heard a rumor way back that they transported individual frames outside of WebRTC; if so, that saves them from having to tie in with that stack.

I really think it should be possible or even easy to write a WebRTC videoframes-only client, I just haven't had any luck. In fact, I've seen a few people saying they simply scripted Chrome with a <video> tag instead! That's madness.

I'm also looking for a fast depth or semantic segmentation model for use in my robotic arm end of arm. Let me know what you wind up using.

Reply on news.ycombinator.com