I recorded a one minute video clip of me editing some code in VS Code (1440p 10fps, using AV1 encoding) and it was about half the size of 60 JPEG screenshots of the same screen. I would be curious to see your numbers if you've done any tests.
The screenshots were taking up a lot of disk space. I noticed that very little changed between pictures, so I started thinking of an algorithm that would make use of this characteristic: store only the changes between subsequent images. A few minutes in, I realized I was reinventing video compression!
So, I just used ffmpeg to turn the image sequence into a mp4. It was something like 95% reduction in file size. (I think I used ImageMagick to embed the timestamp filename into the images themselves, thus recreating basically all the features of TimeSnapper Classic with 2 commands.)
So from on side you have CPU utilization, from the other - network. Network is cheap, but encoding is expensive. This is my thinking at least. I don't have proof - only local experiments, but it's a really good idea to start measuring this.
I also have other ideas in mind on how to scan the screen and send only parts of the screen that have been updated. Probably if I send only a half of the screen, it will beat the video encoding in terms of network. The diff algo should be very fast though, since we're dealing (in case of 1280x720) with 1280x720=914400, 914400*4 = 3.49 MB info processing in 1 second.
Also, curious to hear about video encoding efficiency vs 60x JPEG creation. Is it comparable?
Did you try to use the hardware encoder? Modern computers have chips to accelerate/offload video encode/decode. Your 2019 Mac has Intel GPU for H.264 and HEVC hw encoder, also it has an T2 co-processor that can also encode HEVC video. If you don't supply specific encoders (with _videotoolbox suffix on Mac) via -c:v then ffmpeg will default to sw encoder, which consumes CPU.
> how to scan the screen and send only parts of the screen that have been updated
You'll be reinventing video codecs with interframe compression.
> Also, curious to hear about video encoding efficiency vs 60x JPEG creation. Is it comparable?
I see that you are comparing pixel by pixel for each image to dedupe and also resizing the image to 1280px. Also the image has to be encoded to JPEG. All of the above are done in CPU. In essense you implemented Motion JPEG. Below is a command to allow you to evaluate a more effecient ffmpeg setup.
ffmpeg \
-f avfoundation -i "<screen device index>:<audio device index>" \ # specific to mac
-an \ # no audio
-c:v h264_videotoolbox \ # macos h.264 hardware encoder -r 1 \ # 1fps
-vf scale=1920:-1 \ # 1920px wide
-b:v 2M \ # 2Mbps bitrate for clear and legible text
out.mp4 # you may want to setup a RTMP server so that ffmpeg can transmit the encoded video stream to and allow visitors to view the video
The second version tried to use the VNC approach developed years ago by AT&T and open sourced. The open source implementation glitched just a bit much for my liking. Various companies white labelled VNC in those days; not sure they fed back all their fixes. But the raw VNC protocol has a lot of good compression ideas specific to the screen sharing domain and documented in papers. People also tunneled VNC over SSH. I jerryrigged an HTTPS tunnel of sorts.
After a while I started to suspect if I wanted to get higher frame rates I should use a more modern screen sharing oriented Microsoft-based/specific codec. But it wasn't my skillet so I never actually went down that route. But I'd recommend researching using others screen-optimized lossless codecs, open or closed, so you don't reinvent the wheel on that side of things if you're serious about reducing bandwidth.
Font quality with lossy codecs generally looked too lousy back when I tried it at various settings. Screensharing unlike videos is very text centric.
I'd be willing to do the right kind of lossy but jpeg discrete cosines back then wasn't it.
modern codec do motion prediction and don't work with with low frame rate out of the box.
But could easily do both if you wanted to - though I’m not sure it’s worth the hassle. I agree that this might struggle if used at scale on the same IP
JPEG either makes "smeared" screenshots or low-compression screenshots. PNG often works better.
A proper video codec mostly sends the small changes between frames (including shifts,like scrolling), and relatively rare key frames. It could give both a better visual quality and better bandwidth usage.
What's interesting in the "screenshot per second" solution is that it can be hacked together from common existing pieces, like imagemagic, netcat, and bash; no need to install anything. (Imagine you've got privilege-limited access to a remote box, and maybe cannot even write to disk! Oh wait...)
Just open any website and you'll see text right beside photos, or text against a photographic backdrop, often in the middle of being moved around with hardware-accelerated CSS animations.
I think we need an image container format that can use different compression algorithms for different regions or "layers" of the image, and an encoder that quickly detects how to slice up a screenshot into arbitrary layers. Both should be possible with modern tech. I just hope the resulting format isn't patent-encumbered.
I suppose than the more modern video compression algorithms apply such image analysis already, to an extent. I don't know how e.g. VNC or RDP work, but it would be naural for them to have provisions like that co save bandwidth / latency, which is often in a shorter supply than computing power.
Of existing still image codecs, JPEG XL seems to have the right properties[1]: the ability to split image to areas and / or layers, and the ability to encode different areas either with DCT or losslessly. But these are capabilities of the format; I don't know how well existing encoder implementations can use them.
[1]: https://en.wikipedia.org/wiki/JPEG_XL#Technical_details
Uses a combination of different tech [0]. MS-RDPBCGR is at the base of it all, sort of like the main event loop [1]. MS-RDPEGDI looks into the actual drawing commands and optimizes them on the fly [2]. Then there's the MS-RDPEDC for desktop composition optimizations [3]. Also a bunch of other bits and pieces, like MS-RDPRFX which uses lossy compression optimization [4].
In RDP you don't get to play only with the bitmap or image stream data, but the actual interactions that are happening on the screen. You could say for example that the user right clicked a desktop item. Now send and render only the pop-up menu for this, and track and draw the mouse actions inside that "region" only.
[0] https://learn.microsoft.com/en-us/openspecs/windows_protocol... [1] https://learn.microsoft.com/en-us/openspecs/windows_protocol... [2] https://learn.microsoft.com/en-us/openspecs/windows_protocol... [3] https://learn.microsoft.com/en-us/openspecs/windows_protocol... [4] https://learn.microsoft.com/en-us/openspecs/windows_protocol...
[0]: https://support.apple.com/guide/mac-help/screen-sharing-type...
So, with this, you can have high-quality static text in one region of the frame while there is lossy motion encoding (e.g. for an animating UI element) in another region of the frame.