Web browser remoting using network vector rendering: Cloudflare patent
patents.google.com
patents.google.com
Look if we have enough paid clients asking for this we will definitely do it. Thanks for your comment! :)
- use a modified Google Chrome to capture SKIA draw commands (actually you can just use the DevTools protocol these days, I think the LayerTree Domain exposes usable functionality with regard to this)
- Send the SKIA commands over the wire.
- Recomposite and redraw that on the client.
- Profit (clearly) and save lots of bandwidth.
My product (BrowserBox Pro^0) uses a different method: we just capture screenshots, and send them over the wire.
A couple of years back an ex-Googler from the UK approached me about using the Skia method, told me he had some code I could use, and suggested I use it. I wonder if he knew doing so would have been a patent violation? Surely he wouldn't have suggested it if that were the case, haha! Anyway, I didn't go with it at the time because:
- Complexity of maintaining the Chrome fork, and (if you look at the patent, as I briefly glanced at) it appears running Skia rendering at scale like this is quite complex, necessitating lots of caching, suggesting that parts of that pipeline are expensive, validating my initial concerns about it.
- Complexity of doing the composit on the client. Honestly not that complex, but at the time I was a 1-man shop and the rendering code using screenshots, and the adapting code using network measurements, and adapting the screenshot frequency and resolution was already sophisticated enough and good enough for our customers' uses. We stream it over the faster of WebSocket or WebRTC
- Something about the guy seemed off, honestly, and I was a bit reluctant to work toward some avenue suggested by him because of that.
- I also believed that in the long term, trying to squeeze the most out of the bandwidth (at the cost of code complexity) would be a mistake, because bandwidth capacity is trending up.
- There is an alternative that is probably better than Skia: use ffmpeg to create a video stream out of the screenshot frames and stream that over WebRTC. I haven't added that yet because I value the low resource usage on the server, and adding ffmpeg will change that and add some complexity. But we're willing to do so for any clients that really need it.
- There are some edge cases where Skia chokes with regard to bandwidth: complex gradients, or elaborate text effects. These end up producing Skia commands that are larger than the bitmaps they result in.
- It's plausible that at some point in the future the Skia canvas requirements and the DOM Canvas might diverge, making it not possible to use a simple "clientless" web client for rendering anymore.
These things combined to suggest to me significant tech debt risk in adopting that method, possibly some nefarious risk from a dodgy person, and just the existence of better simpler alternatives made this definitely a no go.
Anyway, I'm posting this here to add to some history and also because it's interesting to me that there is this patent. It's cool there is a patent and Cloudflare is presumably doing good with it. I think the patent is also another reason to not go with Skia, because you definitely don't want Cloudflare acting as a patent troll to come after your nascent startup.
I wonder how this affects folks like Hyperbeam who are also using some sort of "tiled" method to stream the viewport to the client? Or Mighty App--? tho they are probably different as last I looked Mighty App were deep in the weeds with ffmpeg (Sahil was posting his nightstand reading of massive ffmpeg academic tracts, lol!).
If Cloudflare did come after a YC company for patent violation would it be taken care of quietly and a deal made in the background, or would it go the court? Would YC companies get preferential treatment? Anyway, interesting!