Write your own terminal
flak.tedunangst.com
flak.tedunangst.com
- First step (for a UTF-8-input terminal) is interpreting the input bytestream as UTF-8 and "lexing" into a stream of Unicode Scalar Values (https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf#P.12... ; https://github.com/mobile-shell/mosh/blob/master/src/termina...).
- Second step is "parsing" the scalar values by running them through the DEC parser/state machine. This is independent of the escape sequences (https://vt100.net/emu/dec_ansi_parser ; https://github.com/mobile-shell/mosh/blob/master/src/termina...).
- And then the third step is for the terminal to execute the dispatch/execute/etc. actions coming from the parser, which is where the escape sequences and control chars get implemented (https://www.vt100.net/docs/vt220-rm/ ; https://invisible-island.net/xterm/ctlseqs/ctlseqs.html ; https://github.com/mobile-shell/mosh/blob/master/src/termina...).
Without this separation, it's easier to end up with bugs where, e.g., a UTF-8 sequence or an ANSI escape sequence is treated differently if it's split between read() calls (https://bugs.chromium.org/p/chromium/issues/detail?id=212702), or invalid input isn't correctly recovered-from, etc.
> - Second step is "parsing" the scalar values by running them through the DEC parser/state machine.
Unfortunately, you may need to intermingle some logic between these two steps.
While VT100 style control sequences are usually introduced with an ESC, they can also be represented as a C1 control sequence, e.g. 0x84 instead of ESC + D, 0x9b instead of ESC + [. These sequences are raw bytes, not Unicode codepoints, and their encoding collides unpleasantly with UTF-8 continuation characters.
Further documentation: https://vt100.net/docs/vt220-rm/chapter4.html
Since there's no standard which specifies how UTF-8 should interact with the terminal parser, you're a little bit on your own here. But probably the simplest fix is to introduce a special case into the UTF-8 decoder which allows stray continuation characters to be passed through to the DEC parser, rather than transforming them to replacement characters immediately.
As an aside: I wonder how useful it'd be to assemble a report documenting all known terminal control sequences and other behaviors, what terminals they're available in, and how frequently they're used in modern software. There are some big gaps between the DEC documentation, ECMA035/043/048, and actual implementations of terminal emulators.
"UTF-8 still allows you to use C1 control characters such as CSI, even though UTF-8 also uses bytes in the range 0x80-0x9F. It is important to understand that a terminal emulator in UTF-8 mode must apply the UTF-8 decoder to the incoming byte stream before interpreting any control characters. C1 characters are UTF-8 decoded just like any other character above U+007F."
The existing ANSI terminal emulators that support UTF-8 input and C1 controls seem to agree on this (VTE, GNU screen, Mosh). xterm, urxvt, tmux, PuTTY, and st don't seem to support C1 controls in UTF-8 mode. So I don't think poking holes in the UTF-8 decoder is necessary, especially since allowing C1 in UTF-8 mode is rare anyway.
I'm in a terminal most of the day but never have I ever thought of once writing my own. Installing ohmyzsh was a huge step - I'm good for life now.
It is effort, but it's also relaxing. I'll start by saying I don't think this is for everyone and it is far easier if you like minimalist environments. E.g. to me Visual Studio Code for example is an utter nightmare of visual overload. I want my editor to at most show a minimal one-line or less status area and maybe, optionally line numbers. Nothing more. I actively don't want it to pop up auto-completion options or documentation. And so on.
A lot of my requirements makes things simpler rather than harder, and that's not necessarily the case for others.
I started with the editor when I realised my Emacs config was several thousand lines and I though "I can write a whole editor in less". As it happens, a basic editor can be extremely small even in a language like C (e.g. see Antirez' "kilo"), but important: It can be extra small when you write it only for you. I can overlook bugs that doesn't bother me, and only implement functions I care about, and make assumptions about the environment others can't.
E.g. my editor presumes you're running under bspwm. It can run without it, but requires adapting helper scripts, because e.g. "split horizontal" and "split vertical" are implemented by opening a container and starting a new instance of the editor that will fill that area on screen and attach to the same view (the buffers are all held in a server, similar to if you use Emacs in client server mode).
It also contains no code for picking files, because it spawns an external one-liner which uses rofi to select files (same for themes etc.).
After a while, I started thinking about minor nuisances as a result of my terminal. E.g. I like a high degree of translucency, but that makes text hard to read unless there's an outline, so I started hacking outlines into st and kitty, but found other things that annoyed me, and before you knew it I had a "I bet I could replace this" moment.
I don't really care about packing up these for others as-is because so much is specific to how I want things, but I am slowly working my way through and pulling out generic components to release, and then I'll clean up the pase so it's at least there to look at.
EDIT: Just to add that e.g. for the terminal, the entire codebase, including packaging up all dependences other than the standard Ruby library, is currently ~6.5k lines. That includes ca. 1.5k lines for the Toml parser for the config files, for example, so I think it can be trimmed down significantly. For comparison `st` is around 8k lines, excluding dependencies like Xlib, freetype etc, and Xterm is around 88k lines. That's not to crow about how tiny mine is, because a lot of it I get "for free" thanks to Ruby, and a lot of it is down to just not implementing features I don't care about. But it is an illustration of how you can start really small and make something usable when you're free to keep a really tight focus. "Just" the terminal codebase itself is about 1.5k lines. But it will fail to correctly render a bunch of apps that I don't care about because there are still escapes I haven't bothered to implement. That's fine for me, but not for a terminal for general use.
I definitely love that feeling of building just for myself. In the past year I've adopted emacs in hopes to achieve the sort of agency you have to define the experience you're having on your computer.
So I completely understand the motivation, it's just that terminal and text editor seem like daunting projects. I'm SO so busy at work, and after work I need to unplug. I can justify carving out 10-15% of my time to sharpen my axe, but i just assume that a text editor took a team of people years to create.
I think you are highly productive developer. Cheers
That said, while an editor like Emacs is a big project, an editor can start really small. E.g. look up Femto on GitHub, which was my starting point, or Antirez' "Kilo", so named because its about 1k lines.
If you ever get the itch, look at existing small editors and pick one to butcher as a starting point. You can replace every single line over time, but starting from something working let's you focus on what you want it to be rather than how to get it to a point where you can start using it (and even if not something you can use for everything, at least now and again). It's a lot easier to stay with a project like this is you get to where you use it every day, and every improvement makes things a bit easier.
Sometimes I think a shitty visible working prototype is a better goal than a more elaborate deep prototype that takes more time to see results.
for example, I know you can design in a log system, but you can start with printf and do the logging later. It's more important to tickle your motivation brain cells with something that works.
Or, you can start with a working project, strip stuff out, and get something visible. Then add your code.
There's also another (related?) trick - when something is working stop in the middle of a feature that is almost done. The next day you tickle those motivation braincells when you can dive right in and finish it instead of diving into the motivation-killing "planning the next step" stage.
>Mitchell Hashimoto, Hashicorps longest serving IC
Small correction: I don't think this is right. He only became an IC two years ago.[0]
[0] https://www.hashicorp.com/blog/mitchell-s-new-role-at-hashic...
the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements?
the third difficulty is handling font fallback and rendering emojis and other combinatory glyphs correctly.
That diagram you linked is actually quite nice, but visually intimidating. If you think of it as a handful of regular expressions exploded into a state diagram that helps. For instance, the entire left half of the diagram is just the CSI code acceptor, see "CSI (Control Sequence Introducer) sequences" on the "ANSI escape code" wikipedia page.
You can write a regular expression to match a CSI and carry on. This is 2023 and you aren't using an 8080 with 3k of RAM. (probably) The only tiny trick is that you have to handle the "incomplete trailing regex" and wait for more data to arrive and try again.
As for handling reflow. I wouldn't call that an "implementation" problem. I'd call it a "specification" problem. I'd approach it by seeing what Apple's Terminal program does, write that down, and call that my specification.
To switch to/from the alternate screen mode is \e[?47h and \e[?47l. "\e[?" is DEC private mode which are DEC private mode codes. The number specifies a range of settings to switch on or off. The "h" and "l" determines if you're setting or clearing the setting respectively.
The parsing of those are handled by the escape, csi entry, and csi param boxes in the diagram.
You can get something going with just the most rudimentary escape handling just by spitting what programs write to your terminal to debug output in the terminal you run your new terminal from, and add a proper parser a bit later.
You can totally ignore reflow. It'll look ugly. It doesn't matter. When running full screen applications you need to handle width/height reporting, that's all.
Font fallback and nice font handling is a detail to worry about well down the line. There are libraries that can do a lot of the lifting for you depending on language/platform. Just pick a font with reasonable coverage and worry about the rest later.
The answer to this one, incidentally, is ¯\_(ツ)_/¯.
Every terminal emulator engine I've examined handles reflow a little differently. There's no official spec, and everyone has their own idea about how to make it work "best". That being said, I've wrestled with these questions myself, and here's what I've come up with:
* Each line should have a flag on it which tracks whether it should be reflowed with the next line. By default, this flag should not be set.
* The reflow flag should only be set when the cursor leaves that line because a character was printed with the cursor in the last column ("wrapnext" state), and no special modes are set which change the wrapping columns.
* Performing any other action which changes the contents of a line should clear the reflow flag on that line.
The same logic can also be used to join wrapped lines for copy/paste. It won't work 100% of the time, but it will work for most situations.
That's not entirely true. Most serial terminals did have multiple text size modes eg 80x25 but also 128x50 or something. I still have a real VT520 here that can do that. Not a Dec labeled one sadly but one of the later boundless models.
Escape codes are supported, see: https://sciter.com/wp-content/uploads/2022/10/terminal.png
Docs/API: https://docs.sciter.com/docs/behaviors/behavior-terminal
In my case, this was about creating the terminal for EndBASIC (https://www.endbasic.dev/). I wanted to mix text and graphics in the same console, so I had to ditch Xterm.js and create my own thing. It was really exciting to see graphics rendering mix with text “just fine” when I was able to render the first line.
you sure did have to import a lot of libraries, tho
https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu_s...
it's 96 lines of c, which ought to remove any intimidation factor from the task; if i can do it so can you
https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu.h https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu.c
then all the crap to interface with unix braindamage and draw glyphs on the screen, without any actual terminal emulation, is another 232 lines of c
https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu-s...
yeso makes it pretty easy to get pixels on the screen (opposite extreme from 'OpenGL code is very heavy on boiler plate and repetition'), but its text handling capabilities are limited at best
I don't fully understand what the author is saying here or precisely what they mean by writing a terminal in a terminal. From my perspective though, it is easier to write a hosted terminal that runs inside of an existing terminal. Writing the full thing from scratch is a much harder problem. A terminal has many subproblems that are best attacked separately in my opinion.
At its heart, a terminal reads formatted text from standard input and writes formatted text to standard output. It is essentially a REPL. So the first step is to write a (R)ead function. Then you pass the result of read to the (E)valuate function which will process the input and finally pass it in to the (P)rint function. If you start with a hosted terminal, the read and print functions can be modeled with posix read and write so you can devote most of your time to the evaluate function.
Once you have a good evaluate function and the terminal works as you like in the hosted environment, then it makes sense to go back and write new implementations of read and write that target a new host environment. This is when it makes sense to switch to QT or opengl: when you have already implemented the core logic of the terminal and want better io performance. But it also might make sense to target html/js for maximum portability. You can either reuse or rewrite the backend that was used in the bootstrap terminal depending on how it was written. Even if you are changing languages, the rewrite should be much easier than the initial implementation since you already know what functionality is necessary and how to do it.
If you start with QT or opengl, you might never even get to a useful terminal because you get so bogged down in the incidental details.
What I am describing is essentially quite similar to bootstrapping a new programming language. The initial implementation should be done in the most convenient language/environment possible for the author. People commonly make the mistake of implementing a bootstrap compiler in a low level language, which is almost always a premature optimization and forces you to take on accidental complexity (such as memory management) that is secondary to your primary goals. Remember Fred Brook's advice to plan to throw the first implementation away. It is so much easier to do something that you have already done before than something new.
https://github.com/vidarh/rubyterm
It's totally useless for anything other that testing or expanding on, but it was the starting point for the terminal I now run every day, and I'll be updating that repo as I clean up my current version at some point.
The current version uses this for a pure Ruby (no Xlib) X11 client implementation:
https://github.com/vidarh/ruby-x11
And this pure-Ruby TrueType font renderer (I did the Ruby conversion; the C code it's based on was not mine, and is a beautiful example of compact C - look up libschrift) as I got tired of using the bitmap fonts and didn't want to add a FreeType dependency (the renderer is ~500 lines of code):
do you have an output sample like https://github.com/tomolt/libschrift/raw/master/resources/de...?
If it does not produce pixel for pixel identical output it's a bug. But here is the first pass of code that uses the same approach to integrate it with X and use Xrender to output the rendered glyphs:
https://github.com/vidarh/skrift-x11
(EDIT: Fixed the issue below) Note that the example contains a hard coded visual. I need to push a fix for that - it was last updated before I added support to the X11 client to look up the visual. In it's current form it probably won't work for you. Will see if I can push a fix for that today.
i hate x11 visuals and in https://gitlab.com/kragen/bubbleos/blob/master/yeso/yeso-xli... i just assume that everything is 32-bit bgra. i need to get around to fixing that at some point
what do you think an idiomatic ruby binding for yeso would look like? so far i only have lua, c, and python bindings, which are described in https://gitlab.com/kragen/bubbleos/blob/master/yeso/README.m.... my ruby experience is both limited (only a few thousand lines of code) and over a decade ago
X11 visuals are severely annoying. I get they made sense in the 80's and 90's - I spent time in computer labs where the machines ranged from monochrome to SGI Indy's, with 8-bit palette, and 15 and 16 bit direct color displays in between, but I'm all for just assuming 32-bit bgra is available these days.
Any server supporting XRender needs to support it anyway as it's one of the standard formats for XRender. (I don't know if you've looked at the XRender library code, though, but one of the things that made me facepalm was that even though it requires a handful of standard visuals, the new Xrender call that returns the visuals for Xrender does not just return a list of the handful of standard formats as part of the response - you need to filter the full list of visuals and match them to depths and bit masks... Not that it's a lot of code, but it just so aggravatingly pointless)
> what do you think an idiomatic ruby binding for yeso would look like?
I think a bit of a mix between the Python and Lua would get you pretty close. Don't look at "my" X11 client for idiomatic Ruby :) It was "inherited" from an older client, and it's very closely following the protocol more so than idiomatic Ruby, and I'll eventually try to massage it into something nicer.
To take your examples, we'd probably want to use blocks. E.g. I could see the munching squares example looking something like this in Ruby:
def munch(t, fb)
fb.each_row do |y|
fb.each_col do |x|
fb[x,y] = ...
end
end
end
def much (t,fb)
fb.each_row do |y, p|
fb.each_col do |x|
p[x] = ...
end
end
end
Yeso::Window.new(...) do |w| .. and so on; end
Yeso::Window.create(...) do |w|
(0..).each {|t| w.frame {|fb| much(t,fb) } }
end
The class method vs. constructor only really buys you that you can make it slightly harder to accidentally keep the object around since you can prevent it from being directly returned. I think overall that's probably the most common approach even when the constructor supports taking a block directly.
Handly OpenType then adds cubic bezier.
If, on the other hand, you want colour emojis, you need to implement a subset of SVG (though the subset is small).. Yikes.
You get 90% for 10% of the effort... As usual, I guess. Part of me want to see how far down full emoji font support can be golfed, but another part of me feels that's downright masochism.
(and maybe you can do a decent job of blind "hinting" with smarter dsp algorithms, gradient descent, and orders of magnitude more cpu, and just ignore the tt hints virtual machine)
well-defined subsets of svg are a pretty interesting thing; i feel like a smol computing system might benefit from using something like that as its base graphical layer for things like fonts and windows; then a font engine can delegate the rasterizing down to the svg layer
'how far can we golf an svg subset' might be a good rough description of vpri's nile and gezira https://news.ycombinator.com/item?id=10535364
parsing annoying formats is what https://github.com/abiggerhammer/hammer is for; we were able to get it to parse pdf. unfortunately it's not pure ruby. the smallest i've been able to get a peg parser library is http://canonical.org/~kragen/sw/dev3/tack.py which is 27 lines of python, but i'm not sure it can be reasonably extended to do the kind of binary parsing that hammer does without balooning in size
Thanks for the nile and gezira link - it'd be fun to handle the emoji fonts too, at least as an addon...
Hammer looks interesting. I'm right in the middle of yet another parser combinator library in Ruby, but focused on text (it annoyed me that the Ruby Toml parser pulled in a 1500 line dependency for several hundreds of lines of excessively verbose parser definition for a grammar that can be defined in ca 50 lines) and frankly more an excuse to toy with refinements to see if I can get closer to BNF for specifying it.
I might have a look at hammer for inspiration for ways to shrink the ttf parser later.
because i'm not a fan of excessively verbose things, i've been tossing around ideas for a new non-parser-combinator-based peg parsing system called 'the monkey's paw' whose grammars look like this
<expr: <first: <term: <digits: [0-9]+> {parse_int(digits)}
| "(" \s* <expr> \s* ")" {expr}
| <var: [A-Za-z_][A-Za-z_0-9]*> {read_var(var)}> {term}>
\s* <rest: ( "+" \s* <term> {term}
| "-" \s* <term> {negate(term)}
)*>
{add(first, sum(rest))}> {expr}
<expr> {expr}
/where (
<expr: <first> \s* <rest> {add(first, sum(rest))}>
<first: <term> {term}>
<rest: ( "+" \s* <term> {term}
| "-" \s* <term> {negate(term)}
)*>
<term: <digits: [0-9]+> {parse_int(digits)}
| "(" \s* <expr> \s* ")" {expr}
| <var: [A-Za-z_][A-Za-z_0-9]*> {read_var(var)}>
)
based on my experience with peg-bootstrap i suspect i can probably implement this metacircularly in about 100 lines of code in something like lua, js, or ruby, but getting it built in a non-self-referential way will take another 100 or so. i wasn't planning on doing hammer-style bitfields and stuff, and though i might end up with hammer-style parser combinators for bootstrapping, the idea is to use a host-language-independent grammar syntax as the main way to define grammars
I'm really toying with a variety of things. One thing is the smaller/cleaner parsing of binary formats.
The other thing I'm playing with is seeing how close I get to cleanly expressing the grammars in pure Ruby, without parsing an external format. The two things are pretty orthogonal, and always enjoy looking at new ideas for parsing in general, not just what I can abuse the Ruby parser to handle directly...
Your format is interesting. I read it as the <name: ...> bit serving as a capture? If were to map that to Ruby, I'd probably use the Hash syntax to try to approximate it, so you'd end up with something "{expr: {first: ...}". Incorporating the actions without some extra clutter would be a bit tricky, because the operator overload's won't let you add a block, and so you're stuck with a minimum of ->() {...} or lambda { ... }, but I think representing it reasonably cleanly would be possible (of course this looks like a simple enough format to just parse directly as well).
For a taste of how close to avoiding "host language noise" you can get when embedding the grammars directly in Ruby (I've not settled on the specific operators, but it's a bit limited since you can't change the precedence, I do think the '/' which came from ABNF might have been a poor choice and I might revert to '|' in particular), here's a small fragment of the Toml grammar that will parse as valid Ruby with the right code:
toml <= expression & 0*(newline & expression)
expression <= ws & [keyval / table] & ws & [ comment ]
ws <= 0*wschar
wschar <= 0x20 / 0x09
newline <= 0x0a / (0x0d & 0x0a)
The trick to making the above valid Ruby that this is defined inside a class that "uses" a set of refinements[1] that lexically overrides methods on a number of core classes only within the grammar definition, and then it instance_eval's the grammar within an object where method_missing returns an object that acts as a reference to named rules, so each rule is basically just a series of chained method calls (hence the annoying '&' - I think it may be viable to get rid of it by keeping track of state in the object whose method_missing gets invoked, but I'm not sure if it'll be robust enough to be worth the slight reduction in visual clutter)
Without the refinement feature we'd have to wrap the integers etc. if we didn't want to monkey-patch the standard classes and break everything. (This kind of DSL is the only really useful case I've found for the refinements feature so far)
[1] https://docs.ruby-lang.org/en/master/syntax/refinements_rdoc...
And understanding terminal codes which are historically a bunch of kludges on kludges is a guarantee for bugs. I don't think there's any terminal emulator that does it perfectly and the quirks of the most prolific ones like xterm have simply become the ad-hoc standard.