Hacker News Reader

Comment by andrewla | Hacker News Reader

Every time my terminal blows up because I wrote some binary data and I have to run `reset` or `stty sane` I'm amazed that we let things get to this point. Frankly I'm surprised that there are not significantly more vulnerabilities.

Even things like the use of ESC, and the overloading of ESC for the actual ESC key, for a signal for keycodes, and for the ALT/META key are crazy.

I've thought for a while that I should implement an extremely reduced set of escape sequences which use a non-ESC character (maybe SOH or something) for control sequences and key encoding. A minimal set -- color / cursor position / save-screen / query-size -- whatever it takes to make vim/emacs + tmux/screen work, and call it a day.

View on HN

In theory, `reset`/`tset` from ncurses, or an equivalent, is this "thing" that programs are supposed to be using when they need to engage in terminal shenanigans.

Those use TERM/TERMCAP environment variables and some educated guessing reading stdin/stdout/stderr behaviour to figure out what terminal you're using, search for said entry in terminfo/termcap database, and finally invoking native terminal behavior or simulating it with other terminal behavior.

In theory, one could "just implement" an alternative terminal, and put said simplified entries in terminfo/termcap database and things will be all fine.

In practice several of those ANSI escape code behaviors are hard-coded and everyone pretends everything is "at least as capable as xterm" (a oxymoron, given that xterm is one of the most capable and feature-rich terminals around), so you're also gonna have to implement a converter from "xterm ANSI" to your alternative terminal system - sort of how winpty does convert Win32 terminals conventions to ANSI ones.

Then we're back to the starting issue in this converter.

Yes, this is one of the avenues that I've chased down a bit.

One problem is termcap/terminfo itself. The definition has expanded greatly over time, and documentation is poor to non-existent. In many cases termcap/terminfo has been implicitly extended to support ncurses specifically. And like you say, everyone just sort of assumes ANSI/xterm -- nobody has `PS1="$(tput setaf 3)\w $(tput setaf 0) $", you just hard-code the codes.

The last time I chased the rabbit down this hole I got mired in TTY-land and was not able to dig myself out. Too many insane layers, and before you know it you're writing code to emulate serial connections in a vain attempt to make things work without burning the whole world down.

Both (n)vim and tmux have internal TTY emulation as well which makes things even crazier; I think vim uses libvterm and tmux has one internally hand-coded. It's a mess.

tmux, screen, emacs, and vim all have terminal emulators in them because it’s required, not because they’re crazy.

Consider a really simple case. You are using screen (or tmux, or emacs, or whatever) and you have your terminal divided in half so that you can use a program in one half and tail a log file in the other. The program sends a ^L to clear the screen. Screen (or tmux or whatever) must read the ^L and then _not_ send it on. If it sent it on, the whole screen would be cleared, not just the half with the program in it. Instead it has to send whatever control sequences would erase the text from just the right portion of the screen while leaving everything else alone. Virtually everything sent by the program(s) has to be intercepted and reinterpreted in order for it to work right.

No, they're crazy. I already have a window manager I like; I don't need my terminal to implement their own half-assed one, and my multiplexer (running on the terminal) to implement their own half-assed one, and my editor (running on the multiplexer on the terminal) to implement their own half-assed one.

So don't use tmux then. The entire reason most people use something like tmux is because they want a window manager in their terminal.

This is one of the great joys of FOSS, if you don't like a terminal-based utility then you can bet there's a dozen other options available for you to use.

If you really want crazy, run `xterm -ti 340`, then run run an X server from the xserver-sixel repository <https://github.com/saitoha/xserver-SIXEL> in it. Now y ou can run as many terminal emulators, complete with real truetype fonts and all the colors you could want, inside the one terminal. Use a tiling window manager and you’ll be able to avoid using tmux entirely.

  > No, they're crazy. I already have a window manager I like; I don't need my 
  > terminal to implement their own half-assed one,

Basically agree, though note that ssh and dtach and similar tools need to create pty's on the server and client ends because they implement essentially an adapter between a tty and a byte stream.

I agree with your overall point but your example isn't a great one because ^L would be handled by the controlling process regardless of the terminal state. You don't need any special magic here. And blanking specific sections of a screen wouldn't need any special magic unless you're launching other processes and forwarding their output to the terminal. But if you're not forwarding the output of other processes then you can use standard ANSI escape sequences (or even just \s) to blank specific parts of the screen.

\s is a bit hacky for blanking though, it can produce some undesired results in terms of copy/pasting from the terminal and resizing. But if you've got a "full screen" process like you're describing, you'd be trapping SIGWINCH for resize events anyway.

Since I'm getting downvoted so I'll evidence my point a little more:

Bash source for the ^L hotkey: https://git.savannah.gnu.org/cgit/bash.git/tree/lib/readline...

Notice it's calling termcap:

     _rl_clear_screen (0);  /* calls termcap function to clear screen */

So lets look at what termcap does:

    $ tput clear | hexdump -c
    0000000 033   [   H 033   [   2   J
    0000007

Which isn't the ASCII character ^L (0xC) but instead an ANSI escape code (CSI sequence, to be exact) https://en.wikipedia.org/wiki/ANSI_escape_code#CSI_(Control_...

The actual ASCII character for ^L is form feed, (FF: https://www.ascii-code.net) which does serve a similar function but designed for old printers: https://en.wikipedia.org/wiki/Page_break#Form_feed

It doesn't actually clear the screen on modern TTYs. You can try this yourself:

    $ printf "\f"
    
    $ awk 'BEGIN{printf "%c", 12}'
    
    $

Notice how each time it's effectively doing the same as \n?

So in conclusion, ^L clearing the screen isn't something built into the TTY. It's just a hot key like ^U.

There are some hot keys are are driven by the TTY though. ^C, for example. When you press ^C you're not actually sending the 0x3 ASCII character to the controlling process, your TTY gobbles that up and sends a signal to the controlling process instead (albeit this behaviour can be changed by altering the TTYs mode, which requires making a SYSCALL).

So here we've discussed ANSI escape codes, signals and SYSCALLs. However none of these interactions are ASCII characters despite how they're depicted in printed media. Suffice to say, terminals are complicated :)

Amusingly, FF is almost never the control character for clearing the screen. The idea that it is, is widespread, and attributable to some home computers of the 1980s. But it was never the case for DEC VTs. It's equivalent to VT.

Terminal multiplexers like tmux need to create a pseudo-TTYs otherwise they couldn’t multiplex your terminal.

My shell also creates PTYs on some rare occasions too. I’d prefer not to because it’s messy but sometimes there’s no way around it if you want your command line applications to behave like they would on a TTY whilst still having your utility capture that applications output as if it were writing to a pipe.

This is a limitation of TTYs.

The most annoying part of it all is that creating a PTY differs from one Unix-like OS to another. So it’s definitely not a rabbit hole developers go down happily.

> Terminal multiplexers like tmux need to create a pseudo-TTYs otherwise they couldn’t multiplex your terminal.

[Edit: I misread the parent comment as saying multiplexers needed to do terminal emulation. Dtach only multiplexes (using pesudo-TTYs, of course) but unlike screen or tmux does not do terminal emulation. Original comment follows:]

That isn't strictly true; the one I use (dtach) is just a multiplexer.

dtach creates pseudo-TTYs too:

- https://github.com/crigler/dtach/blob/master/master.c#L53

- https://github.com/crigler/dtach/blob/master/master.c#L108

As an aside, that projects source is surprisingly easy to follow. I was able to grok it in just a couple of minutes. Kudos to the author.

Yes, I misread your comment, sorry, having had my mind stuck on terminal emulation.

A PTY is an intrinsic part of terminal emulation.

I think what you’re alluding to is that tmux gobbles up escape sequences (which is configurable by the way) and tracks cell positions. But as I said in another comment, the value add of tmux is its window management capabilities. That’s precisely why people like myself use it. And you cannot have that without complexity. In some edge cases that complexity causes issues, but on the whole I’m more productive for it.

Saying it’s stupid that tmux does that is like saying it’s stupid that motorbikes have gears and fast patrol engines when a bmx works just as well. Sure, to someone who’s never seen a bike before they both look similar. But don’t be fooled by the fact that they both have 2 wheels because they cater to very different needs, hence their very different engineering designs.

> In practice several of those ANSI escape code behaviors are hard-coded

The most common examples I see these days are for red/green/yellow status text in shell scripts. To anyone planning to do that, or even more complex stuff, please consider using tput instead.

tput setaf 1; echo this is red

tput setaf 2; echo this is green

tput setaf 3; echo this is yellow

tput setaf 8; echo this is dim

tput bold; echo this is bold

tput sgr0; echo this is normal

blue=$(tput setaf 4); normal=$(tput sgr0); echo ${blue}Color${normal} text.

(man terminfo for arcane details.)

It's wise to make sure output is going to a terminal before using colors, because they don't play well with logs or text processing commands. You can check with: test -t 1

If you use colors, please consider making them optional, since they don't work well for everyone or in every desktop color scheme. An environment variable makes a sensible switch.

> The most common examples I see these days are for red/yellow/green status text in shell scripts.

Please don't do that without first checking the background colour. Yellow in particular on a light background is unreadable.

(On an xterm-compatible terminal you can get the current background colour with OSC 11 ; ? BEL. Some popular terminals lie about being xterm-compatible, though.)

Make sure one has ncurses installed before switching to tput. Alpine Linux does not install this by default for what it's worth.

# No tput, no cry:

red=$(tput setaf 1 2>/dev/null || true)

echo ${red}text

You tryin' to get Bob Marley stuck in my head? Because it worked.

`tput` is POSIX, though, so you can reasonably¹ expect it to be present.

¹ Ha, ha. The distro I use, Debian, also omits at least one POSIX standard utility by default, even though all of them used to fit nicely on systems with less disk than your current CPU's cache.

Please don't; the possibility of being connected to a non-ANSI terminal disappeared before most of us were born.

Counterexample: https://github.com/nushell/nushell/issues/4096

Old glass terminals are still in use here and there, too.

Win32 console is unfortunately a thing. Not anymore, but was a thing for too long.

And PC/3270. Some very powerful systems like Linux on IBM Z are POSIX and your default "interactive login" choice is PC/3270.

And in the long term, isn't this exactly the problem? Do we really want to perpetuate the status-quo of "I am in a somewhat POSIX environment, so the STDOUT/STDERR if interactive must behave exactly like xterm and if not I will just pretend it is xterm anyways and output garbage."?

Whilst it is true that ECMA-48 terminals are overwhelmingly the norm nowadays, that is not a counterpoint to "If you use colours, please consider making them optional".

Several popular terminals default to calling themselves xterm (e.g. `TERM=xterm-256color`), but are not xterm-compatible, and ignore bug reports (*cough* GNOME *cough*).

Interesting. I don't work on VTE but just landed dozens of patches with no issue at all from pleasant maintainers.

> Even things like the use of ESC, and the overloading of ESC for the actual ESC key, for a signal for keycodes, and for the ALT/META key are crazy.

It actually happened the other way around. The escape character was originally only there as an escape character at the start of a control sequence. It was added to the keyboard back in the teletype era specifically so that the operator could control the behavior of the device on the other end. And it was labeled “Alt Mode”! See <https://upload.wikimedia.org/wikipedia/commons/2/23/TTY33ASR...>, though it is a little blurry.

Terminal behavior dates back to a time when everybody was just making stuff up because there were no established standards or even conventions. What standards we have are mostly just whatever the most popular or longest lived company did.

Terminals are in a bad place because there was a whole lot of effort put towards compatibility instead of standardization. The same curse that email suffers under. So the "standards" end up hopelessly complex and also terribly vague because they're trying to support every dumb thing every single company did at the same time. And the worst part is that most all of those companies would cease to exist in short order afterward, so the system is chock full of bug-accurate support for long dead systems that literally nobody uses anymore.

Well, yes and no. ECMA-48 is an established standard, and dates from 1976. The standards in use today have been around that long, long enough even that terminals that aren't ECMA-48 compatible are exceedingly rare nowadays.

So it's really the reverse of what you state. There has been a lot of effort put into standardization, and compared to the 1960s we are highly uniform in the 21st century and widely adhere to an almost 50-year-old standard. It's not the long-dead systems that spoil this. As evidenced from the headlined article, it is the still active systems adding things like vendor-specific commands for setting titles and clipboards and colours and displaying images and whatnot that have introduced the complexity.

A strictly ECMA-48 only emulator doesn't have even half of the problems mentioned in the headlined article, because the control sequences mentioned are extensions to the standard introduced by various GUI terminal emulator programs.

> and the overloading of ESC for the actual ESC key

What else was the ESC key made for if not the control code?

> I've thought for a while that I should implement an extremely reduced set of escape sequences which use a non-ESC character (maybe SOH or something) for control sequences and key encoding. A minimal set -- color / cursor position / save-screen / query-size -- whatever it takes to make vim/emacs + tmux/screen work, and call it a day.

I did that as a 8086 boot sector and it would render tmux with unicode characters stripped. You need surprisingly few sequences.

On a modern PC, Esc key is Esc key... whatever the application wants to use it for (99% time, cancel some UI operation).

To me it's insane that one of the most used key command has a 1 second delay in the terminal.

Or that typically Esc + some function/arrow key results in some junk.

If you type in junk into the keyboard, junk is what you get on the screen. I see the fault with the developers that overload the keys for existing control characters with a different meaning.

> whatever the application wants to use it for

With the application being a terminal emulator, that is emitting an ESC character.

> one of the most used key command

Found the vim user! The vast, _vast_ majority of Unix command-line or text-based programs do not use the Esc key because they were not written with a particular terminal or keyboard layout in mind. (Unlike vim.)

But for whatever it's worth, I use vim every day and I notice no 1 second delay when switching modes with the Esc key, so perhaps this is something you can configure in your terminal.

Ctrl + [ instead of Esc gets one back into normal mode, with no delay Takes a while getting used to it (I still haven't)

What does `Ctrl + [` send to the terminal for you if not esc? In my experience these are always the same, so Vim shouldn't be able to tell the difference unless it's a GUI-integrated version like gVim.

Try to run `xxd` and then press Ctrl-V and either Ctrl+[ or Esc, then Enter and Ctrl+D. Typical terminal setups will show the same codes both times (1b0a for esc and \n).

Alt+Space (or Alt+J Alt+K Alt+B to go up, down, left one word from current position after returning to normal mode) is what I often do which should work universally.

  > I notice no 1 second delay when switching modes with the Esc key, so perhaps 
  > this is something you can configure in your terminal.

This is configurable in Vim. ttimeoutlen, timeoutlen, and noesckeys are pertinent. tmux et al may also need tweaking.

Is your terminal emulator waiting for a second input before sending Esc? That’s not a common behavior AFAIK.

No, its waiting for input after sending escape. Escape + character is how you reach the C1 control codes, with CSI and OSC (ESC + '[' and ESC + ']') being popular ones (both referred to in the posted article). This is relevant for keyboard input (as contrasted to output-only) since some codes for special keys (like arrow keys) start with CSI.

Edit: You asked about the terminal emulator, but its the console application that has to do the wait.

Are you using something like tmux: https://unix.stackexchange.com/questions/608142/whats-the-ef...

> Every time my terminal blows up because I wrote some binary data and I have to run `reset` or `stty sane` I'm amazed that we let things get to this point.

Every time that your terminal emulator does this in response to binary data, wonder why your terminal emulator hasn't taken the very sensible route that was taken by the authors of mosh years ago, to simply not support alternative GL and GR character set switching, treating it as something whose time has passed given that we live in a UTF-8-to-the-terminal world nowadays.

I don't know what magic combination it is but I keep hitting something on WSL2 and Windows Terminal that makes my session freeze into a blinking cursor. This results in me having to close the current tab and start a new terminal session completely.

We're still inventing new ways of breaking arguably the most basic user interfaces.

control-Q and control-S freeze/unfreeze the output stream (can't remember which is which). May be that?

It doesn't seem to be that, I can't even reliability reproduce it. I originally thought it was something with the right control or right alt, but it's not consistent.

I keep doing this in gnome-terminal, and for years I was confused about what was going on, I thought it was a bug, but I found out what it is! It turns out there is a "read-only" mode, with some keyboard shortcut that I was triggering by accident. You just have to right click in the terminal and uncheck "read-only" to get it back.

Dunno how long it's been since you used it, but I believe keybindings for this are disabled by default now and you have to toggle them on in your profile preferences.

Probably CTRL-S, which is the software XOFF flow control. If so, CTRL-Q sends XON, which will un-hang the session and resume data flow.

Ah yes, CTRL-Q, the most intuitive keybind for resuming a frozen terminal after pressing CTRL-S, the least likely combination to accidentally be pressed while typing.

It's mnemonic! [S]top and [Q]ontinue :o)

[Q]ontinue was the next best option, c is already taken up to [C]ill the current process obviously.

Reminds me of the game NetHack. How do you consume a potion? Well, you [q]uaff it, of course!

They really should have switched the latter with Verti[k]al Tab.

CTRL-whatever does something funky often enough that I only hit CTRL on purpose.

Stop/Qontinue.

x & 0x1F is what the Ctrl Key used to do, with x being the other keypress. This made the ascii control characters accessible using the Control key (hence the name). So you can access Tab with Ctrl+I, the newline characters with Ctrl+J and Ctrl+M, the backspace with Ctrl+H and also XON and XOFF with Ctrl+Q and Ctrl+S.

They weren’t chosen with mnemonics in mind, because there are 32 of them. You might be interested in the keyboard on the IBM Model 33 teletype (<https://en.wikipedia.org/wiki/File:TTY33ASR.jpg>) which had labels for many (but not all) control characters directly on the keycaps.

That’s fascinating. I’m embarrassed to have been making fun of our elder gods without even realizing it. They really were gods for creating such devices, given the constraints at the time. I doubt I could’ve done much better.

The most interesting part is that we still use the same keybinds by default 60 years later. Defaults matter a great deal, but one does not expect a 60 year example.

Cheers for this. Have a wonderful weekend!

There’s really no need to be embarrassed; we do a really poor job of teaching people these things.

Honestly, it’s basically all path dependence. Features get added all the time, but we just keep layering them on top. A lot of the features that already exist are still useful, so none of them can be removed.

IBM was never the manufacturer of the Model 33. It was designed, made and sold by the Teletype Corporation.

Huh, you’re right. I’ve been misremembering that for years apparently.

This can be disabled.

This has JUST started to happen to me recently a few months ago, when sshing into a server and running screen with an emacs inside.

When I reconnect and restart screen back into emacs, ^S pauses output instead of doing an incremental search.

I have to pop out of screen with the right escape sequence to get to the login shell I ran it in, then go "stty -ixon" to disable XON/XOFF flow control. Or always remember to do that before running "screen -RD".

But I never ASKED for flow control, and it has worked PERFECTLY for decades without ever getting into +ixon mode, until just a few months ago, and now it happens every single time.

Did something change recently, is ubuntu auto-installing earth shattering patches that maliciously change the default tty settings behind my back (because XON/XOFF flow control suddenly became cool again?), or did I take a walk on a foggy night and accidentally slip into an alternate universe where it's always been that way for everyone else, and I'm about to be seized by silent men in white coats and taken asylum which turns out to be a rehab clinic for people who can't read minds?

https://en.wikipedia.org/wiki/All_the_Myriad_Ways

I finally gave up and put an "stty -ixon" in my .bashrc, but I get the ominous feeling that's one of those tempting but dangerous kludges you're not supposed to do -- Isn't there a superstition about never touching your tty in an rc file? Or is that never touching your pp in a login file? -- that can come back and bite you in the ass later when you least expect it.

It seems like this is just the latest in a long series of cascading ass bites.

Terminals in UNIX systems are anything but basic. Windows terminal takes it to the extra level by supporting two OS terminal models.

Terminals in non UNIX, that not pretend to be serial printers, like the Amiga DOS, or plain MS-DOS, those are the basic ones.

Windows might even support 3 if you count cmd and powershell separately :lolsob:

cmd and powershell are shells, not terminals.

The Amiga terminal is simple in some ways, but it supports stuff most Unix terminals, won't, such as the ability to shift the text positioning by individual pixels in both x and y direction, which was of course promptly "abused" to draw bitmap graphics (exceedingly slowly) to the terminal.

MS-DOS had ANSI support with ANSI.SYS [0], that support never got into Windows NT's CMD.COM.

[0]: https://en.wikipedia.org/wiki/ANSI.SYS

Yeah, but that is a device driver, not part of the command.com itself.

You are conflating and confusing several different things, including what operating systems even had a terminal paradigm. Windows NT started out with a "console" paradigm, like OS/2 before it, not a "terminal" one. And command processors like CMD are not what provide the "console" parts of those systems.

"Every time my terminal blows up because I wrote some binary data and I have to run `reset` or `stty sane` I'm amazed that we let things get to this point."

Rather than reset or stty sane, I use something like one of the following, based on what software is available, e.g., there is no reset on NetBSD

  echo w1b63|ired -n /dev/stdout

  echo 1b63|xxd -p -r

  echo -e '27\n99\n'|nc-data -g

  printf \\33c

Yes, the terminal is an embarrassment. It's telling that Sixel is state-of-the-art. A format from the 80s designed to support dot matrix printers.

Sixel isn't state of the art, see https://sw.kovidgoyal.net/kitty/graphics-protocol/

I'm aware of that, but it's supported by a total of 4 terminals.

Hmm, which terminal? I remember catting binary files to the terminal used to hose things up pretty good, but any decent terminal emulator seems to have made that not a thing in recent years. I just did `cat /bin/*` in Konsole and it left me with a perfectly good prompt.

If you use fish, that should never happen. (If it happens, we would consider it a bug.)

(Author here.)

Unsure how fish can fully protect from this, as if you run "cat" in fish, then it isn't between you and the terminal anymore? Or do you mean fish should do a reset before displaying each prompt? If I run the printf commands from the "flash.c" example in fish I end up with a messed up terminal.

fish maintains its own tty state and pushes/pops it each time control is returned to fish or fish shells out to another process.

That's still only best effort, for example it will depend on the terminal but it doesn't reset things like xterm's control keys send escape sequences property, so:

  printf "\e]4;1;?\a\e[>4;2m"

In xterm+fish will result in a messed up shell which you can't escape from (as ^C to clear the input doesn't work anymore).

Correct, it only handles prevents the basic kind of borked that can be reset with `stty sane` or `stty reset`.

Reply on news.ycombinator.com