The terminal escape sequences ocean is deep and dark | Hacker News Reader

The terminal escape sequences ocean is deep and dark

ethanheilman.com

100 pointsEthanHeilman59 comments

I used to write a lot of telnet/ssh games back in the day. Telnet has a bunch of command codes as part of its protocol. Each command starts with an IAC code. This code could be anywhere within the stream of bytes coming from the end-users terminal. IAC is \x01b\x0ff\x0ff. The fun part of implementing text-based network protocols is you must scan, byte for byte, until you reach a \n or whatever your line-feed is. As you are processing the stream, if you come across an escape, you must branch off into the command processing loop instead of the input processing loop. Likewise, responses to these commands can come AT ANY TIME. Mid sentence from a user? Yup. Randomly as you are sending your buffer? Probably. Full-Duplex mode with ACK? Not a guarantee.

When developing terminal services, you're standing on the shoulders of giants and must account for 30+ years of terminal hackery. The up-side is you can also use that same terminal hackery for fun things like progress bars, emoji's, colors, blinking (ux faux pas) for attention, spinners, tables, and yes - even matrix rain effect by positioning cursors and clearing partial screen coords. [1]

[1] https://github.com/gabereiser/go-rain

Ansi escape codes are fun tho! My favorite so far's been finding a way to scroll the screen up or down by one line [1] in a way that works on as many terminal emulators as possible - the spec says how to do it, but most actual terminals don't implement it correctly.

Another fun one was asking the terminal how big it is [2] (and also figuring out the modem speed as a side effect) -- especially when a major telnet client fails as soon as the width is above 256 characters [3]...

Both of these would have been simple if the terminal is a local one, but the fun part is dealing with an unknown implementation at the other end of tcp - and the fact that that's even possible :-)

[1]: https://github.com/9001/r0c/blob/master/r0c/ivt100.py#L1651-...

[2]: https://github.com/9001/r0c/blob/master/r0c/ivt100.py#L625-L...

[3]: https://github.com/9001/r0c/commit/5e7d64d7f81cab3350259b0cd...

r0c is a really cool project. I should dig through this codebase and see what tricks I can learn.

When terminals were real physical devices, including paper, accidentally streaming binary to them was really interesting. It got pretty noisy. Maybe it's a corollary of zipfs law that ^G is going to appear in any binary stream more than you would think.

The modern experience of a laser emitting sheets long after you think you've killed the rogue non-PPD non-PDF input stream, is analogous.

1) $ cat /vmlinuz | write $VICTIM

2) Laugh as your friend's terminal goes audibly berserk, drawing the attention of everybody in the computer lab.

3) Apologize for getting your friend in trouble with the printing office.

Put 'mesg n' in your shell startup file to prevent that.

We had an old IBM dotmatrix printer in our computer room back in the day, which the operators loved to warn us would actually catch fire if we didn't get our CR/LF vs LF situation solved on some of those 100-page reports that management always needed to have printed on Friday afternoon, when nobody would be around.

One time, I forgot to add the check to the queue, and I got aroused from my Friday hijinks with the alarming call that the Halon system had gone off and our operator wanted my blood. The printer had actually caught fire, due to 20,000 missing LF characters putting every single line on the same character (carriage return), burning a hole in the platen and yeah .. setting off the Halon.

That's when I learned, as a junior programmer, the value of pipes.

> That's when I learned, as a junior programmer, the value of pipes.

Sounds like a perfect BOFH episode.

I knew a few BOFH's in those days. It was truly a wild and woolly time for us junior programmers, conducting the necessary voodoo to gain access to the "aquarium", getting a chilly terminal assigned either far away in some dark corner under the Halon horn, or .. even worse .. right next to the Day Operator, whose shoulder-surfing capabilities induce trauma even now, thinking about it .. good times, good times.

The day I got my own pizzabox and ethernet connection, and thus didn't ever have to bother an Operator for access to anything, ever again, was another big step up in the career path I chose for myself .. still, I miss those grumpy old fuckers. They had good stories to tell, usually, and I still 'sync;sync;sync' to rewind tape, now and then, by muscle memory ..

I think you have a few years on me then. When I was an undergrad, we had a slightly elderly vax 11/780 (iirc) and a line printer that probably deserved 3-phase power.

Oh, this was the mid-80's and the Vaxen were still around, being replaced by stuff from WANG and Tandem .. but that line printer was sacred for some reason, it stayed in operation (after repair) until the late 90's ..

actually, in most places wang got replaced by vax and later by pcs.

Not in my experience. :P

wang went bankrupt in 1992, while the vax line soldiered on for another 10 years or so. anyone that replaced a vax with a wang misread the market badly, particularly with the introduction of pc wordprocessors.

What is the collective term for a group of Wangs?

Is it Wangers? A wank? Management? Parliament/senate?

We used to say "wangen", as in "hey wangers, did you shut down the wangen?", which was properly used a source of much mirth for a few years ..

> There are an many types of escape sequences. I've yet to find even a complete listing of all of them. No terminal supports all of them.

It's impossible to even find them all; there were no standards (until the ANSI controls were standardized quite late in the process) and every terminal mfr (most of whom are long dead) made their own. Eventually many started by copying the VT100 set, because DEC was a big company, but even then they'd implement a subset and/or add more. And of course there would be namespace collisions. So yes, no terminal ever supported all of them, and that wouldn't really make any sense.

As for soft terminals (terminal emulators): just pick a terminal (the old hardware kind I mean) that's in the termcap database and implement only what it does. ANSI is a good default.

(none of this makes the author's problem any easier; just expanding on one part)

That's why we have ncurses and terminfo. The work has been done, and there's a good library out there to give you the ability to write readable code that works across multiple terminals.

ncurses and terminfo are a nightmare, and many decades out of date and unmaintained.

When I wrote my own terminal emulator and client it was easier to just start from scratch.

Just support a reasonable subset of xterm and telnet. You don't need to delve into support libraries for physical terminals that haven't existed for decades.

> Just support a reasonable subset of xterm and telnet.

That doesn't work, though. Not reliably.

> OSC-1337 is used to copy files to a remote server

They couldn't have chosen a better 4-digit number for that one.

Probably intentional :-)

one of my more interesting jobs (for some values of interesting)was programming an ibm 7171 protocol convertor that took the 3270 codes that our ibm 4381 super-minis spat out and converted them to ascii sequences that our serial terminals could understand, and the reverse for things like cursor keys coming from the ascii terminals.

this involved writing tables using a limited regex-like language, a bit like termcap/terminfo on unix. you could actually load a revised table (these things were difficult to debug) while the 7171 was running, and even sometimes get away with it, without crashing all user sessions.

like all ibm kit of that era (mid 80s) the thing was beautifully built and documented

It’s all been downhill since physical bells were removed.

Yes, it is! Also, terminal multiplexers are pretty annoying if you actually care about what escape sequences do, since they have to parse them all and either act on/answer them themselves, or pass them along to the terminal. Possibly with modifications.

`dtach` is just a pass-through multiplexer, with no emulation of its own. For my use case — reconnecting from the same local windows to the same remote sessions every morning — it's perfect. I can use the full capabilities of my terminal rather than the intermediate subset.

It's basically HTML+CSS but with text and without the enormous amount of documentation and tutorials.

I guess the real question is whether a new method could be used that would provide a greater benefit.

I'm loathe to say this, because every couple of decades or centuries someone does the impossible, but until that happens:

It's impossible. There's too much legacy stuff, it would be like boiling the oceans, and there's both no real glory to be had and no money to be made.

Oh, and for the legacy stuff. Since there is no real API, everyone did as they pleased, for the new protocol/standard/tools to take off they would either have to be amazing and almost completely supersede existing stuff (ultra hard) or to offer backwards compatibility, which the lack of the API would make it a humongous software undertaking. I've been watching what the Oil shell is doing to just extend Bash in a smart way, and the amount of work going into just that is mindboggling.

You'd need a madman (more likely, several) like Lennart Poettering and look at what did, it got him death threats for writing software and getting it adopted.

> it would be like boiling the oceans

Well, it's kinda happening, finally, right? The oceans and atmosphere heating up? So not impossible.

I've spent sometime thinking about this. My too much coffee hot take is that you'd want to separate the presentation layer out and provide much structured ways of interfacing with the processes. It would look something like this:

You'd have your high level CLI which would call programs, those programs would provide:

1. input and output in form of JSON services. This would function like a modern pipe functionality,

2. a CLI UI which talked to those services. This CLI can swapped out for another at the callers digression,

3. a built in debugging and code reading system similar to developers console in modern web browsers,

4. and hooks for intercepting calls made outside the program either to the OS or the dynamic libraries like LD_PRELOAD

In some sense it would replace the terminal with a web browser like CLI. Unlike a web browser you'd be able to mix and match CLI UIs and pipe processes together.

I'm not convinced that is a good idea. The beauty of terminals and the power of being about do stuff like process substitution largely comes from the use of simple abstractions. Would a JSON based pipe be less or more powerful than pipes as they are today? I suppose you could always use a utility to strip the JSON and turn it into the newline text streams that the commandline today is so good at manipulating.

The more serious issue is that this library apparently does not enforce a length limit on escape sequences.

It seems to have a hand-written ad hoc parser, handling untrusted input. A very rich source of exploits. (Luckily it's at least using a memory safe language so what you can do is likely limited to DoSing yourself or using up all memory as in this case.)

I know this sounds sarcastic, but I don't mean it that way. Are there virtual terminals which are not hand-written ad hoc parsers?

They seem to be using this library for some sort of remote shell session snooping. At the very least this sort of negligence lets an attacker smuggle commands through the system unnoticed.

An assumption made by many virtual terminals is that the input and output stream are trusted. For instance `cat file.txt` in a terminal result in your terminal being sent escape sequences that will change how your terminal functions well after you quit cat.

I've always wanted a virtual terminal that had a terminal state stack, when you run a command or open a program, all the changes to the terminal are pushed on the top of terminal stack. When I quit the program or the command stops, the state is popped. It seems like it would be very difficult to implement such a thing in the current terminal framework because as far as I can tell, they do not understand when a command starts or stops. Then again the effectiveness of virtual terminals over the last 5 or so decades is likely because of this simplicity and its corresponding flexibility.

A few terminals (e.g. iTerm2) try to track the current command, presumably by looking at the children of the shell they spawn. This isn't much use when your current command is `ssh`.

You could probably have a push/pop escape sequence requiring a matching unique token that intermediate output can't guess.

    OSC 57473 ; 1 ; <token> ST
    ⋮
    OSC 57473 ; 0 ; <token> ST

Can bash do this? I wonder what issue I have yet to think of prevents pop/pushing escape sequences from being default behavior.

I think with most current shells you could do this in prompt strings. Actually, you'd probably want a combo that restores the top-of-stack state and leaves it there; then you'd push the state during shell initialization and restore it in the prompt.

OMG my pal Mike and I went seriously down the rabbit hole doing this for DOS in the last 80s/early 90s. We went seriously, needlessly baroque and it was GLORIOUS.

My bash prompts have all been dead boring by comparison.

Step 10 of the walkthrough as an error. A -> C

Thanks, good catch!

Fixed it in the blog.

I have to say I'm impressed you read that far and found my error. I enjoy the fact that I can post something technical like this and someone would read this much into the details. I debated writing up my debugging notes. I might try writing up more of my debugging notes.

Made my day, thanks!

seeing this last night led to me reading the original VT100 user manual. Wow, documentation was great in those days.

what will it take to get rid of all this legacy and create something new for today's needs? is it even remotely possible to do so?

Whatever you want will not age well.

(Remember the XML craze a few years back? Probably not, and that's a good thing.)

It can age very well, that's not the problem.

If it's designed by a small team for this specific use case, not to take over the universe.

But then the question becomes, who assembles this small crack team, and for what?

Why would they get paid? Who would pay them?

This is like basic research, more or less. Nobody wants to pay for it, but everyone would want now the benefits of the resulting awesome offshoots 20 years into the future.

Windows does it differently, with explicit functions to call from what I remember.

>Typically they start with the byte \u00b1 called the ASCII ESC character. ESC is the 27th ASCII character.

27 ..... b1 ..... I am amazed this person wrote an entire article and never once noticed the absurdity.

I don't follow, whats the absurdity?

No absurdity, just typo. It should have been 1b, not b1.

The absurdity that an upper 8-bit ascii range character would be an elementary control character in a terminal.

The absurdity that a number less than 32 would start with b in hex instead of 1.

The absurdity of making this mistake consistently in a post about diligently fixing bugs and not proofreading or checking it.

How does one not notice this?

Weirdly enough it is flipped in actual pull request text too but not in pull request's code

> How does one not notice this?

coz people skip funny magic numbers when reading

>How does one not notice this?

Trivially easily?

> The absurdity that an upper 8-bit ascii range character would be an elementary control character in a terminal.

Wait 'til you hear about the C1 control set (§5.3 in the current version of ECMA-48).

> Gen z programmers, not even once.

I know I shouldn't even reply to this, but I'm old enough that I have used physical 8-bit ASCII terminals with C1 controls (and ISO 2022 page switching).

I wish I had the ability to subconsciousnessly convert hex to decimal when scanning a page. Sadly I have to intentionally do math in my head and that involves telling myself to look for such "absurdities". Thankfully my blog is just a git repo so I easily push fixes.

Thanks for the feedback.

27 is way to small for a byte value that starts with the hex digit B.

You aren't wrong, but that requires asking the question is this hex value correct? There are mistakes I can see without thinking such as missing semicolons and there are mistakes that require me to mentally tell myself "check this value". I would love to get to the point where my brain notices that I flipped a hex character automatically, but I rarely have the chance to work with bytes and hex values directly.

If I see 345,345,345+123,123,123=578,578,578,578 I probably wouldn't notice that this is incorrect but once I decide to really look at it the three errors are obvious.

As I commented below, it's incredibly striking exactly because it betrays a complete unawareness of the logic behind these systems.

These aren't just funny codes, there is a whole mindset behind it. ASCII being grouped in chunks of 32. Being able to flip between uppercase and lowercase by flipping a bit. The history of upper 8-bit codepages in DOS.

And I didn't even comment on the fact that your "byte" uses 16-bit notation.

Reply on news.ycombinator.com