There's a demo program that comes with libcaca, called cacademo. It can either output text to standard output with ECMA-48 control sequences, or crank up a built-in X client that displays a very simple text window.
I SSHed into a remote machine with it from a local machine that had an X server. Tunnelled over the SSH connection, the X version of the demo was a little slower than running directly on the remote machine's own display, but mostly didn't drop frames. The mode where it sent ECMA-48 control sequences over the SSH connection to a local terminal emulator was woefully slower, to the point that it was displaying less than 1 frame a second sometimes, and visibly tearing the display. I tried it with both Microsoft Terminal and MobaXTerm to see whether it was the terminal emulator. It wasn't.
Sending colour information as ITU-T T.416 RGB control sequences, or even as AIXTerm 16-colour control sequences, to be interpreted by a control sequence state machine in the terminal emulator and turned into a display buffer and graphics drawing commands, turns out not to be as efficient over SSH as the X protocol over SSH. Yes, X11.
The X path was going through an X client library to the X server. It was probably using the old X11 primitives for text, rather than doing client-end rendering. The terminal emulator path was going through ncurses, then probably wasn't being batched up very much over the SSH connection, and then was going through a decoder that had to (amongst other things) parse numbers from human-readable to machine-readable thousands of times per frame because ECMA-48 uses decimal-character-encoded values in control sequences, and the cacademo program outputs lots of colour changes, frequently a colour change for each successive cell.