you seem to be attempting to define 'terminal' as 'vt100 emulator', which unfortunately condemns you to only unreliable solutions like sixel that are too inefficient to provide smooth graphical interaction and full of compatibility problems. you can plot a graph from a remote machine, sure, but if you want to use your mouse to zoom in on it, you just bought a one-way ticket to jank city

but i've used graphical terminals like ncd x terminals since 01994, and they provided smooth graphical interactions even over shared 10-megabit ethernet. current xorg and xwayland do a great job of emulating those terminals, with most of their deficiencies fixed, and ssh has built-in support for transparently passing along the needed credentials

xpra works even better, supporting h.264 compression and reattaching to applications you've detached from

the big problem with x11 is that trying to write a program to do something simple like display an image or draw a bar graph ensnares you in five pages of bureaucracy

to solve that problem i wrote yeso: https://gitlab.com/kragen/bubbleos/blob/master/yeso

here's a program to display a png with yeso:

    #include <yeso.h>

    int main(int argc, char **argv)
    {
      ypic img = yp_read_png(argv[1]);
      if (!img.p) return 1;
      ywin w = yw_open(argv[1], img.size, "");
      yp_copy(yw_frame(w), img);
      yw_flip(w);
      for (;;) yw_wait(w, 0);
    }

it also runs on the linux framebuffer without x-windows

here's a color generalization of munching squares in yeso in python:

    import yeso


    def munch(t, fb):
        for y, p in enumerate(fb):
            for x in range(fb.size.x):
                p[x] = (t & 1023) - ((x ^ y) & 1023)

    if __name__ == '__main__':
        with yeso.Window(u"CPython μunching", (256, 128)) as w:
            t = 0
            while True:
                with w.frame() as fb:
                    munch(t, fb)
                t += 1

it uses python context managers so you don't forget to yw_flip()

there's also a luajit binding

the example programs above are non-interactive, like virtually everything sixel that isn't full-screen; here's a lua program to test mouse interaction:

    local yeso = require 'yeso'

    function bounds(a, b)
       if a < b then return a, b else return b, a end
    end

    yeso.Window("mΩuso", {256, 256}):run(function (w)
          local m, dragStart = {x=128, y=128}
          while true do
             for ev in w:events() do
                if ev.isMouse then
                   m = ev.p
                   if ev:button(1) and dragStart == nil then
                      dragStart = {x=ev.p.x, y=ev.p.y}
                   elseif not ev:button(1) then
                      dragStart = nil
                   end
                end
             end

             w:frame(function (f)
                   f:fill(0x7f7f7f)

                   if dragStart then
                      local minX, maxX = bounds(dragStart.x, m.x)
                      local minY, maxY = bounds(dragStart.y, m.y)
                      f:sub({minX, minY}, {maxX-minX, maxY-minY}):fill(0x336699)
                   end
             end)

             w:wait()
          end
    end)

as for embedding interactive graphics in a text stream, i'm still trying to figure out the bubbleos solution to that. but i'll get there, and the result will be usable over ssh, while sixel probably hasn't gotten there even for low-bandwidth interactive localhost cases and probably never will