Sixel is defined for two purposes: the first is to allow display of bitmaps. the second is to allow the definition of characters glyphs (which xterm does NOT have -- due to graphics limitations). If the glyph definition could be done, your bandwidth requirement would be reduced. Back in the 80's we generated custom fonts for PostScript (the Apple laser printer PostScript would cache the glyphs). As needed, the printer would request glyps and these would be supplied by an external driver or box (we called this the "Robin Box" for fairly obscure reasons). This system provided pre-press proofing for many customers (which would include publishers like Ziff-Davis). Please note that the speed of the communication was 9600baud (1000 cps). The image of that printer was 300x300 dpi, with some printers doing 600x600. Sixel would have been good... we used ASCII.
I like sixel -- I use xterm which gives me the option. For everything? No. but if I need a graphic, it is easy enough to use. The alternative for most would be to simply generate PostScript, and run GhostScript to view the results (typically on another system). I have NEVER contemplated watching a video with sixel... I could do it, but the player would be a "labour of love" - I would use a common decoder to produce an uncompressed bitmap (scaling and colour reduction) then convert that into frames of sixel... Just to show StarWars on a 340. But, no urge.
The main issue is that sixel offers the feature. No real reason to NOT have it... and it exists.
Try pbmtoln03, ppmtosixel, imagemagick convert and lsix
sixel is not a protocol: just an encoding. Your idea conflates the two things.
The purpose of the Robin Box? The publishers typically had hundreds or thousands of typefaces. The PostScript Printer? 15 to 50. Since RAM in the printer was limited, this approach allowed the scanning of the target typeface, conversion to outline form and production of dynamic programs executing in the printer. That are discarded but results cached. (and note that technology is within a period that, in this case, was bounded by RAM space, and typographic conventions -- all of which changed fairly rapidly).
That approach allowed hundreds of typefaces to be used on a single page! With "standard" PostScript; on a printer with only a megabyte or so of RAM available. Sixel is simply a similar tool. You really can't predict how something like would be used if it were generally available.