Anatomy of a Terminal Emulator
poor.dev
poor.dev
In particular, I like his focus on experimentation to figure out a reasonable upper bound on performance (so you can measure success/failure), "non-pessimisation" (don't do things extrinsic to the actual task), understanding the actual machine and its capabilities (he doesn't use the term, but sometimes called "mechanical sympathy"), and tactics for isolating "bad code" that you probably have to use (at least to get started).
I like the conceptual separation of the two (real) optimization techniques into: "hardcore optimization" (my own term) which is intensive, careful measurement and fine-tuning; and this idea of "non-pessimization" which is the principle of doing the least possible amount of work while delivering the required features, analyzed from a "basic algorithms" and "fuzzy mechanical sympathy" perspective. "Hardcore optimization" is inordinately expensive so can only be used sparingly, but "non-pessimization" is much easier and should be used thoroughout your codebase. His argument is that just applying non-pessimization alone represents a 100x-1000x speedup compared to typical code. And in particular if a codebase is pessimized, it's largely pointless to do hardcore optimization because every time a new section is optimized it just reveals countless other sections that are hopelessly unoptimized and become the new bottleneck. I.e. non-pessimization is a prerequisite to effective hardcore optimization.
I also like the idea of isolating "bad code" as he puts it, but I feel that term is a bit uncharitable and implies an unnecessarily narrow scope. I don't think "bad" in the sense of a general value judgement is the operative term here, and it would be better stated more flatly as "slow code" or maybe "expensive code" because it would apply as equally well but to more situations. For example we'd still like to cache the output of a library that is written as well as could be expected for what it does but is still too expensive compared to our required throughput.
Caching with a key derived from hash(concat([input bytes],[output byte #])) is a pretty clever idea to multiplex the hash table, but I don't really love that it has to hash the input bytes multiple times, and it doesn't solve the problem of figuring out how many output glyphs it should look up. I suspect that adding an output length parameter to the cache entry, and using a glyph cache that supports outputting multiple glyph sizes (even as separate arrays of power-of-2 sizes, 1,2,4,8,16) would have been less pessimized. :)
HashValue = HashRound(HashRound(NothingUpSleeveSeed, BytesCount), Bytes)
He calls this “pelican hashing” which is presumably some reference to the hashing literature, I’m not familiar.
The program is very cringe but it was my best attempt at the time as I couldn't find any literature on terminal emulators (what actually happened was that I didn't know what to search for)
Anyways, I've successfully pivoted my career away from economics and into software development which was the goal when I took the job in college.
Those were the days!
I may have fallen out of love to suckless.org, but the code is usually simple so one can at least learn from it.
As much as I sympathize with cutting things down and making frugal software I am too messy to make it all work for my advantage. I now prefer to mix and match and not attaching myself to any group. But I am fond of the actual code. It is refreshingly simple and easy to track. I was using dwm the most from all of the projects and I still think of going back to it. Though currently I'm getting tired of software all together.
No. Here in France they are sometimes organized as local events, sometimes they're used to --- surprise --- protest. We also have "descente aux flambeaux" (downhill skying with torches.)
It's also the naming of their mail server after Hitler's hidden base in Poland, and the railing against "cultural marxism" (itself a Nazi trope). Taken together, these things should register as "yep, probably a Nazi" even if one of them alone wouldn't.
Most recent example - Charlottesville?
More on topic, this does seem like a decent overview.
Here though I don't think it's the case.
While the overall focus of the book is on programming with Notcurses[2], the author shares a wealth of related info and history throughout its pages.
I'm 99% sure it's inserting ANSI escape codes (I'm maintaining a terminal emulator myself, and really that's how everything works), but I could of course be wrong.
I am thinking that .foregroundColor and .backgroundColor do the same thing via legacy emulation in conhost.
Each screen location was two bytes, one byte for the character value, and one for the character attributes which were a set of bits that controlled red, green, blue and intensity for both the foreground color and background color.
Then you could write routines that would fill in a rectangular region with a color, scroll the text of a region up or down, and do all sorts of other windowy things. There were also interrupt routines you could call that did some of these, and certainly routines in Crt that did some of this. Then you were on your way to developing your own TUI library!
Alternatively, you would issue an interrupt call to put the screen into 320x200 256 color mode, get the address of that buffer ($B800 if memory serves), similarly overlay it with a typed grid, then start poking byte values in and getting all sorts of nice colors out of it. Super fun!!
I looked recently to see if an equivalent of Crt is still out there, but it doesn't look like any of the modern Pascal versions support it.
Short version: there is support for ANSI and VT sequences in the Windows console (which relatively recently got substantially expanded), but that's not what it speaks "natively"-- for Win32 console applications, there's a native console API that works by passing IOCTLs back and forth between the app and console driver.
(If you're writing a terminal emulator for Windows, you don't necessarily see this-- the new ConPTY mechanism you use to build these as of Windows 10 abstracts away the Windows specifics so you see text and VT sequences just as you would on *nix.)
Or is there some voodoo by which the ANSI sequences get stripped from the output when redirected to a file?
Have you double checked with s hex editor what is in the txt file? Suppose it could also be your text editor that doesn't want to render those codes.
Not quite, but programs in general use some voodoo to detect when they are being redirected and won't output ANSI codes when they detect that.
Often there will be a flag to enable/disable color, or let it detect when color is desired. On Linux ls accepts the --color=WHEN parameter, where WHEN can be always (`ls --color=always > ls-with-ansi-codes.txt` will output color codes), it can be never (just don't output ANSI codes, no matter whether you detect output redirection) or it can be auto (will show colors when you execute `ls --color=auto` but not when you do `ls --color=auto > ls-without-ansi-codes.txt`).
Often times libraries that let you specify output color will helpfully query the capabilities of the output device, and avoid writing the control codes if it’s (for example) a plain text file.
On posix platforms, there's an API to check if a given file handle is a tty or not. https://github.com/corasaurus-hex/isatty/blob/main/isatty.c is an example of using the API, in a Janet context.
I assume that .NET's support for Linux/macOS is using that API decide if it should strip color codes or not.
So its not just porting a terminal. It's the entire OS. Of course there is p9p, plan 9 port, which is a port of the core plan 9 user space tools to Unix systems. It does offer a draw server that can be mounted.
Part of the challenge is to find capabilities to add that are sufficiently simple and compelling to use in command line apps vs. as a web app or native GUI app that'd be more widely accessible than a new terminal capability.
Really, I think this mostly requires imagination from those of us developing for this platform. We have all the capabilities we need. We just need to create apps that don't require you to look up cheat sheets and man pages to use, and provide powerful alternatives to existing GUIs. There's a bit of a renaissance in this area as of late, but I feel this is just the beginning.
I use my own text editor. I expect to remain its only user, and that's fine. It's not intentional. It just isn't a priority for me to make it usable for anyone else (though I do separate out parts of it into libraries / gems as functionality matures).
As such, I'm looking for improved terminals not to make things that doesn't require cheat sheets (because to use my editor you'd have to be comfortable with reading the source). It's fine that others want other things, but I think this is part of the challenge with improving terminals: A lot of the user base are advanced users who want to improve their own workflows, not write something user friendly, and who wants tools that are easy to combine and chain, where the priority is not something user friendly.
The balance is fine in that if you're looking for something particularly user friendly in the sense of friendly to less advanced users, targeting a GUI framework or the web is very quickly going to provide a better experience.
EDIT: I saw zellij mentioned in a sibling, and looked at it, and incidentally it's quite similar to my setup, except I use bspwm and scripts to manipulate the tiling so that e.g. doing a vertical or horizontal split in my editor results in a new editor instance in a new tiled window rather than in a new pane in a single terminal. That way I can treat the few GUI apps I use exactly the same way (mostly Chrome). I'd love it if there was a standard API to split panes and start apps in a given pane, so both terminal multiplexers and tiling wms could be targeted with a single standard mechanism.
My experience has taught me that tools become better the more people use them and are able to participate in their creation and maintenance. The more user friendly a tool is, the more users you'll have. Personally, I think a tool can be very user friendly and still powerful enough for advanced users.
I used a very similar setup to yours before I started Zellij! The reason I started the project was in order to be able to formalize such a setup (for me it was implemented as a soup of bash scripts which I dreaded moving to another machine, not to mention handing to another user).
One of the things we want to do with Zellij is port it to the web and maybe even in the future to use it as a sort of "backend" to power tiling window managers. I'd be totally open to do this in a standardized way if maintainers of other tiling managers are game.
I've taken the approach that rather than try to shoehorn my use into a full app, I aim for my text editor to be as small as possible, by reusing external tools whenever possible, as I do agree with you that it's worthwhile to have as much as possible of the code used by other people. But at the same time I realised there are editors smaller than my old Emacs config. So e.g. my editor relies on bspwm to split panes, and on rofi (or anything that can take a list of things to choose from and return the chosen thing) to select files or themes or buffers, Rouge for syntax highlighting, and anything that I can make generic enough I'm splitting into gems (the editor itself is written in Ruby). The way I see it, I want the editor to be a tiny little core that's mostly configuring other components. Currently it's about ~2.6kloc, but much of that is code that can be split out or will disappear as I clean some things up. I don't want it to get much bigger than that - preferably it'll get smaller.
> I used a very similar setup to yours before I started Zellij! The reason I started the project was in order to be able to formalize such a setup (for me it was implemented as a soup of bash scripts which I dreaded moving to another machine, not to mention handing to another user).
The big limiting factor for me with something like Zellij over my current setup would be having it work alongside e.g. Chrome and the occasional other gui app.
> One of the things we want to do with Zellij is port it to the web and maybe even in the future to use it as a sort of "backend" to power tiling window managers. I'd be totally open to do this in a standardized way if maintainers of other tiling managers are game.
If you provide a mechanism that allows a client to split panes already, then maybe the easiest starting point is for someone to just pick a config location/format tools can look for a command line to do splits in. E.g. on bspwm, the command given to split horizontally might simply be sh -c 'bspc node -p east ; exec #{cmd}' &. On i3wm, the same would be i3-msg 'split horizontal; exec #{cmd}'. Currently my editor just blindly executes "split-horizontal re --buffer numeric-id-of-the-buffer", and I have a "split-horizontal" script in my ~/bin. It'd be trivial enough to have it read a config file to find out what command to execute instead.
Especially when my daughter is taking a course on Unix and the instructor is superficially touching in these fundamental topics (due to their own limited understanding), I am going to use this to teach the underlying concepts.
E.g., why is it called a terminal emulator and not a terminal user interface? Why is there a "y" in pty if it stands for pseudo terminal? Why is it call it a "pseudo terminal" if it's acting like a communication pipe?
The y in pty is because of tty.
- The original terminal was a teletypewriter, printing onto paper, connected to a computer, usually through a serial interface. A teleltype is abbreviated "TTY", and the original Unix terminal interfaces were given the device file names /dev/tty, /dev/tty1, /dev/tty2, ... Those were the hard-defined terminals. Yes, these were communication pipes, handling stdin (keyboard), and stdout and stderr (the printer).
- As psuedo terminals came to be used, for users connected via "glass TTYs" (CRT terminals such as the venerable VT-100 and VT-200), or by entiirely virtualised terminals through remote connections (telnet, rsh) or windowing systems (W and X11), pty came to be used for pesudoterminal. Again, these were communication pipes. I used a variant similar to this VT-320, notable for its amber phosphr display: https://yewtu.be/watch?v=RuZUPpmXfT0
Yes, historical quirks.
You can still hook Unix up to a teletype, by the way: