Terminal graphics protocol
sw.kovidgoyal.net
sw.kovidgoyal.net
A lot of the power of the terminal metaphor is its simplicity and universality. You can implement one as a purely mechanical device (as has been done numerous times), or as an application running on your graphical environment. With a GUI terminal window you can run the very same applications you'd run on a ASR-33 teletype (although the sounds and smells won't be present).
There's actually this rather common expectation that the end user is the one that is in charge of personalising their system to their exact needs.
People need to run apps via uart or any other kind of remote link that might come in the future.
There was a graphical terminal called the Blit (sold as the AT&T 5620, succeeded by the DMD 630 and then the 730 "Multi-Tasking Graphical Terminals"). They implemented windowing, multiple streams, graphics and a lot more on top of serial connections. Essentially, multiplexing streams on top of "a UART". What you describe is more or less what they did back then, minus the windowing and multi-tasking functions, because we let the host OS deal with that.
There is even a neat 5620 emulator for macOS you can download from the app store. It requires a proprietary shell running for that tty, and I never took the time to figure out exactly how it works.
https://archives.loomcom.com/3b2/documents/DMD_Terminal/ has some good docs on that, and bitsavers.org has a ton more on the 630 and 730.
I hope one day someone finds a ton of them forgotten in a warehouse in the middle of a desert in the US, because, where I live people recycle old computers with an uncanny dedication (those barbarians!).
They decided it was a bad thing (also compatibility) and now they implemented in-band ANSI escape sequences and recommend them.
But also the difficulty of running a terminal session outside the traditional Windows command line app (think VS Code terminal pane).
https://devblogs.microsoft.com/commandline/windows-command-l...
That does not mean it is efficient. If the data is on the GPU then with in-band signaling you have to copy it off of the GPU to send it to the terminal and then the terminal has to copy it back to the GPU.
Can't we define what behavior should a modern terminal have with these already existing tools instead of inventing a brand new wheel?
Pixel aspect ratios are not always 1:1. Fonts have character height to width ratios which are all over the place and a terminal which must adapt any character cell to any region of an image pixel perfectly is way more difficult than it needs to be. Unless you're Casey Muratori, probably. then there are things like display scaling, which, ideally, the terminal application itself shouldn't need to know anything about in order to render correctly. What good is a terminal that renders pixels exactly if the OS can then magnify the window so that you lose the pixel "perfection"?
I don't like the Kitty protocol described in this post, either. The Windows Terminal people are thinking about this, believe it or not, and they are the ones that convinced me that existing methods are insufficient. We need yet another standard for doing this. One that can adapt to video display technology of the 1990s, at least.
> I don't like the Kitty protocol described in this post, either. The Windows Terminal people are thinking about this
I've been part of these discussions (mostly from the VTE side). I agree sixel, ReGIS, and Tek are "legacy" and have been implemented in such ways it's a miracle that they ever worked, and what you see on one brand of terminal is entirely dependent on the positions of the planets when the code was written and when the image is rendered and might not resemble at all what you'd see on another brand unless by coincidence ;-P.
Which is kind of why I brought up DPS - it's the only one that takes pure geometry and generates pixels (Tektronix does so as well, but it's very limited, and doesn't really like to be anchored to the text screen).
Very different from VT100-compatible communication, though, but maybe it can be encoded to fit that mold.
It reads like what you get when someone has a poor idea of what 2D UI requirements are (alignment, relative positioning, layout, out-of-band image delivery, updates, animation, ...) and then has to retroactively fit them in one by one.
And then the mechanisms it has for doing so read like they are a poor fit for how you want to use them, requiring weird global bookkeeping/state, and sound like a nightmare without a proper API around it, abstracting it away.
Aka reinventing all the lessons of 30 years of UI development from scratch.
From what I remember sixel is _worse_ but that doesn't really change things.
Your only actual complaints were "weird global state" and "no API". I look forward to seeing you design a protocol that involves two way communication between two unrelated entities that avoids some shared state between them. As for no API, this is a protocol specification. if you need an API so badly go browse NPM.
Nothing is wrong with them: Sixels for example work perfectly for all the tasks I throw at them in my terminals, from showing pixel perfect images to playing videos with mpv.
> Can't we define what behavior should a modern terminal have with these already existing tools instead of inventing a brand new wheel?
Actually, something is wrong with these already existing tool: they are preexisting, and therefore someone else idea. If you don't invent a new wheel (on which you can take credit for), you have to recognize that 40 year old technology is now perfectly adequate.
I think it wasn't ideal in the 1980s due to the bandwith requirements, but in 2023, if I can play videos with mpv, I think that's enough for most use.
I think the tech has been adequate for some time (at least about 15 years), and I can freely recognize that both because I have no horse in the game, and I care more about actual use than having/creating a shiny new tech.
Inventing a better wheel is the perfect way to avoid confronting such issues: you can argue your new wheel is better, and technically it will be true, even if in practice it's no different that the old wheel, in the sense it allows your vehicle to go forward for all the practical measures you care about.
There may also be a loss-aversion effect: if what you invented that was great is no longer needed, it may be painful to recognize it. You may not want to go for what's technologically not as nice as your invention, but that can now do the job just as well as your invention did, thanks to Moore's law.
Some people also just want to see their favored pet idea to win - unfortunately, they pet idea is only theoretical, or if it exists, it has a very small install base: since sixel is "adequate" or "sufficient", blocking/sabotaging sixel support is the logical action to prevent it from becoming established through network effects.
Put all these people together, add a little handwaving about how old formats can't work, and you get the current situation.
BTW here's a project I'll try to resurect: rending X applications ... in sixels: https://github.com/csdvrx/xorg-sixel
So you could have graphical applications in your terminal! I've just up. I've only uploaded the picture as I'm still fighting with the code but what you see is Xeyes show inside wezterm thanks to the sixel output of Xorg.
For terminals that don't support sixels (yet), sixel-tmux will convert them "in flight" into derasterized graphics: see https://github.com/csdvrx/sixel-tmux/
I'm sure this tech stack (X apps -> Xsixel -> sixel-tmux -> text) will make many people cringe, but in terms of functionality, it works (latency, bandwidth etc are sufficient for my uses) and if you have sixel support in your terminal, you get pixel perfect results.
FWIW I use and love Hyprland, but I want to use portable graphics apps more than I care about the underlying tech being better.
Terminal Graphics Protocol - https://news.ycombinator.com/item?id=35940691 - May 2023 (3 comments)
Legend of the Red Dragon even supported RIP.
0. https://mitchellh.com/writing/ghostty-and-useful-zig-pattern...
In other words, there is practically speaking a well established alternative, and the remaining task is to badger all the holdouts into switching.
And I think it's a little ignorant to suggest that the issue is lack of an alternative. How about just don't use it? Or take a photo of your own kid and upload it
Then of course it's not pixel-perfect unless you make your terminal very large (like 800x240 instead of 80x24) but something being better than nothing, I'd argue it's for the better if all you can do is 80x24 with no pictures otherwise.
The Lenna photo, originally taken from a Playboy magazine, perpetuates a narrow and objectified view of women. Its continued use in tech demos and educational materials not only overlooks the rich diversity of alternative images available for such purposes but also subtly endorses a culture that many in the tech community are striving to move away from.
In an era where inclusivity and sensitivity are rightfully gaining prominence, it's crucial that we reassess and update our educational and demonstrative tools to reflect these values. Opting for more neutral and universally appropriate images would not only avoid potential discomfort among a diverse audience but also demonstrate a commitment to fostering a more inclusive and respectful tech culture.
Just my two cents.
"I retired from modeling a long time ago. It's time I retired from tech, too... Let's commit to losing me." -- Lena Forsén
Put the Linux mascot. Use the Windows XP background image. The Monalisa or the Sixteen Chapel.
Of Pirate, Cameraman and Peppers, only Peppers seems to remain:
https://sipi.usc.edu/database/database.php?volume=misc&image...
Or I'm being thick.
(Edited to add: Mandrill is still there, but the paper doesn't recommend it.)
https://github.com/PLEX-GR00T/OpenCV_Tutorials/tree/master/I...
https://commons.wikimedia.org/wiki/File:Mae_Carol_Jemison.jp...
This is a NASA photo so it's in the public domain in contrast to the copyrighted Lenna image.
> Although Playboy is notorious for cracking down on illegal uses of its images, it has decided to overlook the widespread distribution of this particular centerfold.An astronaut? Some would see that as classicist. Blue and Orange? This impacts people with deuteranomaly and protanomaly color blindness. NASA? They are a capitalist space program that hired Nazis after WWII. A woman? Sitting? A person of color? Rectangular dimensions? ect... ect...
These are, of course, completely absurd objections. But someone somewhere would complain about your choice as firmly as you complain about the Lenna photo. And would you think their objections are as absurd as some think your objections are? Probably.
Standards are standards because they provide a solid reference that everyone can use and they should, by definition, be resistant to change. If it ain't broke, don't fix it.
BUT if we do change it, I vote for a cute cat photo. https://www.photos-public-domain.com/2018/04/26/cat-with-blu...
Just use a different image. It's a pretty simple ask. For fuck's sake, this shouldn't be so hard to understand with a tiny shred of human empathy.
This is ignoring that the photo itself is completely innocuous. People who complain about the Lena photo, I suspect that 99.9% of them have absolutely no problem with AOC's appearance at the Met Gala in her Eat Mor Chikin dress. Which, from a perspective of propriety, there is no possible differentiation from the Lena photo. Their complaint has to do with the photo's origins, not with the photo itself. And, sorry, but the world isn't ruled by the pearl clutchers on their fainting couches.
https://www.nbcnewyork.com/entertainment/the-scene/met-gala/...
It's like asking Americans to use metric.
We don't live in that era. We live in an era where you are either "with us" or "agaist us".