ANSI escape codes (2019) | Hacker News Reader

ANSI escape codes (2019)

notes.burke.libbey.me

145 pointsbrendanfalk82 comments

The article nor the comment section mention curses or the terminfo database, which makes me sad.

Wanna read something about terminals and terminal control sequences? https://invisible-island.net/ncurses/terminfo.src.html

Disclaimer: i use a DEC VT520

Because just like the code page in locales, you should not need to care about terminfo if all you want to do is show some basic color. In $current_year it is OK to assume that the current terminal (if stdout/stderr is a TTY) can at least parse color escape sequences - if yours doesn't then create your own wrapper that strips them. Generally the way git does it should be considered best practice. If

- Output is a TTY

- The NO_COLOR environment variable is not defined

- The TERM environment variable is not set to "dumb"

then enable colors by default.

The git approach here is not "not caring about terminfo", but delegating the caring (of the difficult parts at least) to another program, namely the pager. The default pager `less` depends on the information in TERM and the terminfo database, and it has to, since functions like toggling to the alternate screen buffer are not standardized per ansi.

git works without a pager too.

I agree that you will need something like terminfo if you writing a TUI like a pager, but for most CLI programs you only need simple color output and maybe cursor movement on and erasing of the current line and for that terminfo is an unnecessary dependency.

Line editing is tricky enough that i would recommend a thing like the readline library, which in turn uses ncurses.

Just for colored output alone, i agree.

In my horrible opinion, you can have a fairly portable terminal application without using the abomination that is libncurses, just straight escape codes / control sequences.

Main reason is that most of current terminal emulators are actually xterm emulators: they support a good subset of xterm features. Also, the exceedingly strange console that was in Windows 95 is finally out. What quirks and differences are left, can easily be detected at runtime, no need for $TERM and terminfo/termcap

> the exceedingly strange console that was in Windows 95

That only ever tried to be a terminal emulator when ansi.sys had been loaded. But it's been gone for quite a while now. The Windows console host on the other hand only gained terminal emulator capabilities in somewhat recent Windows 10 versions.

This is my korn-shell fragment that I use to set up color codes. Everything but the final print is posix-compliant (change it to a printf to run under bash).

    N='\033[' x=30

    for a in Bl R G Y B M C W # 4-bit Black Red Green Yellow Blue Magenta Cyan White
    do eval $a='$N'"'"$((   x  ))"m'" \
           b$a='$N'"'"$((60+x  ))"m'" \
        ${a}bg='$N'"'"$((10+x  ))"m'" \
       b${a}bg='$N'"'"$((70+x++))"m'"      # bX=bright Xbg=background bXbg=brgt bgnd
    done

    N=$N'0m' x=0

    print $bB'This is bright blue.'$N

This is one of the many cool things that zsh comes with out of the box. It ships an autoload-able colors¹ script that does largely the same thing as your fragment, but you know it available wherever zsh is.

In much the same way it comes with both a termcap² and terminfo³ module, allowing you to easily avoid hardcoding system specific escapes in to your config.

¹ https://github.com/zsh-users/zsh/blob/master/Functions/Misc/...

² http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-...

³ http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-...

Agreed.

We made two decisions early on: xterm is the only terminal protocol, and UTF-8 is the only encoding. This has spared us unending amounts of pain, because we don't abstract over these things. There is a bit of divergence between xterm implementations, but for what we do, not much: if you want to print images there are a few strategies, none of them universal, and the closest (sixel) isn't great imho.

We've also found that, with a modern terminal, a lot of what curses brings to the table isn't necessary. While we did divide the terminal up into zones, and only repaint changed zones, we shouldn't have bothered: as long as you hide the cursor when you start a paint, and show it after you put it back, filling the entire screen with a complex/colorful xterm pattern happens within one refresh, so no flicker.

There is certainly no need to calculate a minimal delta change and only print that, on a reasonably modern system (anything from the last ten years will do).

This.

There is no reason to use terminfo or libncurses in 2021.

How would you check if a incoming escape sequence is the F1 key pressed by the user?

Just assume your the terminal is a vt220, a.k.a. xterm.

The funny thing is that termcap and vi were not created for a DEC terminal.

They were created specifically for the Lear Siegler ADM-3A, which predates the VT100. The work was done entirely by Bill Joy, who later joined Sun Microsystems.

https://jbcrawford.us/history/computers/adm3a

It seems like all the remaining terminals and emulators have gravitated to the same feature set so people just assume it doesn't matter? There was, at one point, a fair amount of really oddball terminals, like the Hazeltine 1500 that weren't VT1xx compatible. And they were at least minimally useful on a Unix box with terminfo/termcap.

In theory, yes. In practice, gravitating towards the same feature set doesnt imply that it will end up with the same sequences (code for the F1 key):

  # for i in linux rxvt-unicode xterm; do TERM=$i tput kf1 | hexdump -C; done
  00000000  1b 5b 5b 41                                       |.[[A|
  00000004
  00000000  1b 5b 31 31 7e                                    |.[11~|
  00000005
  00000000  1b 4f 50                                          |.OP|
  00000003

> ...like the Hazeltine 1500 that weren't VT1xx compatible.

Don't forget the 1500 came out before the VT100.

Your comment is a reminder of how dominant DEC was at the time, the Apple to IBM's Microsoft.

The VT52 predated the 1500, though I imagine there are earlier Hazeltines.

Indeed, both of these terminals were part of a Cambrian explosion of glass TTYs.

I mentioned the VT100 specifically as that was what the comment referred to. Also because for some reason it somehow became iconic, though I don't know why (I always thought the VT52 superior, though the best IMHO was the Ann Arbor Ambassador).

> It seems like all the remaining terminals and emulators have gravitated to the same feature set so people just assume it doesn't matter?

In the context of Unix-like systems, that is pretty much true. But, broader than that context, there are still terminal emulators in use which are completely unrelated to DEC VT. People still use IBM 3270 emulators to talk to IBM mainframes, and IBM 5250 to talk to IBM i (formerly known as OS/400). People still use Unisys T27 terminal emulators to talk to Unisys mainframes.

There are also some terminal emulators out there with unusual features. Consider AccuTerm, a terminal emulator specifically designed for use with PICK systems. If you read page 207 onwards of [0], you will see escape sequences to display images, run commands on the client machine, trigger file uploads and downloads, and even a facility for an escape sequence to embed a VBA script which is then executed. A lot of that is quite dangerous if you don't trust the system you are connecting to, or if the system you are connecting to doesn't correctly filter escape sequences out of untrusted data; but, assuming it does, it can be used to easily create some integrations between legacy PICK apps and Windows desktop apps such as Microsoft Office.

[0] https://static.zumasys.com/zumasys/atfiles/manuals/8/AccuTer...

Having been recently delving into getting a TRS-80 DT-1 emulating a Televideo 910 working with Linux, I couldn't agree more.

Even now for modern devices it wouldn't be the worst idea to avoid hard coding things like color (e-ink?). And it's not like it's difficult to avoid hard coding, using tput ( https://linux.die.net/man/1/tput ) means you don't have to write out the arbitrary escape sequences yourself while being compatible with any terminal.

"My team writes a lot of command line tools" is a valid excuse for internal use. But not for writing a educational post.

> And it's not like it's difficult to avoid hard coding

There are upsides to hard-coding the escape sequences that you aren't mentioning. Using terminfo or ncurses requires an extra dependency with a clunky interface. Using tput requires spawning a process, which gets slow if you have to do it repeatedly. But simply hard-coding the escape sequences is dependency-free, as fast as printing, can be done in any language, and is going to work for at least 99.9% of the users out there.

I'm usually a big fan of "use a standard interface rather than one particular implementation", but I haven't found a solution, other than hard-coding the ANSI escape code set, that I'm happy with.

You'll also find a lot of Google search results for "tput: No value for $TERM and no -T specified". I wish it was as simple as just calling tput, but you need to write a bunch of logic anyway.

> Using terminfo or ncurses requires an extra dependency with a clunky interface.

Are there any linux distros for any platform that aren't shipping terminfo? I could see some embedded systems cutting it down for small storage solutions. But I don't think it's unfair to assume that terminfo is installed. ncurses is more complex so I won't disagree with that one, but tput isn't part of ncurses.

Clunkiness is an opinion, I see needing to look up what "\x1b[7m" means as being clunky compared to "tput rev".

> Using tput requires spawning a process, which gets slow if you have to do it repeatedly.

If you're pretty printing text for a human to read then I don't think the couple extra ms it will take to spawn a handful of tput processes is going to be an actual performance concern. If you're logging data that's another thing, but this was specifically about CLI programs.

> can be done in any language, and is going to work for at least 99.9% of the users out there. > I haven't found a solution, other than hard-coding the ANSI escape code set, that I'm happy with.

This is most likely true and I don't disagree that my stated use case is in the 0.01%. I was going to refute the any language comment with libraries based on terminfo, but all the ones I ran across in 30 seconds just hard code the ANSI escape sequences. Since it will likely work 99.9% of the time, it is "easy" in a way to hard code them. So I understand why it is being done that way and why you would also prefer to hard code them. But there is a standard interface to use, it just may not be perfect.

This may be something where we are technically stuck between a rock and a hard place. We have a good resource database of technical information on how to format text. But there isn't a significant reason to put in the effort to use it.

> Are there any linux distros for any platform that aren't shipping terminfo?

Ah, I was referring to higher-level languages such as Ruby here (I have a lot of scripts where I use it a shell script replacement) rather than C, where you can just download a compiled version and use it without any fuss. Ruby moved its curses interface out of the standard library a long time ago, so now you have to add it to your gems list.

> If you're pretty printing text for a human to read then I don't think the couple extra ms it will take to spawn a handful of tput processes is going to be an actual performance concern.

For scripts, I agree — I used to use tput, and only stopped because I'd eventually seen the ANSI escape codes enough time to just memorise them. But I find that if your program needs to use enough colours (my software has been called "overly colourful" before), the time it takes to run a tput once for every bold and normal variant of all eight colours quickly makes it seem like your program is running on the JVM, even if I cache the output. I appreciate that not all software needs every terminal style available to it, but it seems to me that the best interface should be able to scale to many styles, not just a few.

> I was going to refute the any language comment with libraries based on terminfo, but all the ones I ran across in 30 seconds just hard code the ANSI escape sequences.

This doesn't surprise me! I used to be concerned with the growing number of comments and guides on the internet that say things like "this is how you do bold in a terminal" as though it applies to all terminals, so I asked a question on Stack Exchange a while ago:

https://unix.stackexchange.com/questions/548158/in-2019-is-i...

I was hoping to find an answer from someone like yourself, who regularly works with non-ANSI terminals and was becoming increasingly annoyed at the proliferation of standards-shirking shell scripts... but I only got comments saying I was doing it wrong, as though I was the only one who didn't get the memo. You're the only obscure terminal user I've ever talked to in the wild, congratulations.

Anyway, I agree that we're stuck in this situation, and even though I am taking the easy way out, I still find that a shame.

Funny, I write terminal codes directly, because where I learned to program (PC DOS), there was no terminfo, only ANSI.SYS. I don't ever plan to port anything to DOS ever again, but old habits do die hard.

"Having been recently delving into getting a TRS-80 DT-1 emulating a Televideo 910 working with Linux, I couldn't agree more."

That sounds very familiar. I swear some guy put a video up on Youtube attempting the same thing. :)

Love the channel. Keep the videos coming.

I deeply regret that I only have one upvote to give.

Back in the 90s I played a MUD called Ancient Anguish. It's the type of MUD where inventory is dropped on logout. I learned they didn't filter for control sequences, and so I would rewrite the text on screen to do all sorts of nefarious things, like make people think they were ambushed and almost dead (thus possibly trying to quit to save EXP loss). I could also forge tells/yells to make them look like they came from someone else. It was funny to me at the time, lol.

edit: for fun i just checked out it's current status and it's still around, awesome. and rule #4 is now don't abuse control codes. hahaha, sorry guys. AFAIK they started filtering them soon after me and a cohort caused some chaos.

There is much more, though.

See here for an overview:

https://github.com/albertz/wiki/blob/master/terminal-escape-...

There are a couple of non-standard extensions, e.g. by iTerm:

https://iterm2.com/documentation-escape-codes.html https://iterm2.com/documentation-images.html

Some of them are pretty complicated to standardize. E.g. see this discussion on simple image support:

https://gitlab.freedesktop.org/terminal-wg/specifications/-/...

I've been learning kitty the past few days. It also has several extensions (image mode, more complete keyboard handling, and a few others):

https://sw.kovidgoyal.net/kitty/protocol-extensions.html

Here is an overview of image modes:

https://github.com/kovidgoyal/kitty/issues/33

The terminal hyperlink thing seems quite useful. Kitty has some magic where if you click a hyperlink over a direct ssh connection it shows you a menu that has some options like download or download/open in editor/reupload when done. I haven't checked the details yet how it determines that you are in an ssh session.

On a Linux box, you can also just do

  man console_codes

which is significantly more extensive than the posted link

For those who aren't on a Linux box at the moment, the online version of that man page is at https://man7.org/linux/man-pages/man4/console_codes.4.html

Thanks for that. I’ve mentally added it to my list of useful man pages not related to a specific command (`man ascii` and `man hier`). I see the `manpages` package provides a few more manuals describing computing concepts in Section 7, e.g., `cgroups`, `charsets`.

To the 4 of you who actually care enough about this information to write something with it I offer you the following blatant self promotion:

oho https://github.com/masukomi/oho is the best ASNI->HTML converter out there.

useful if you want to take your colorful terminal output and share it on a web page, OR if you're really geeky you can write CLI tools with colored output, to summarize data then convert it to HTML, then to PDF and share it with your coworkers. (Chrome's command line interface is the best HTML to PDF converter I've found).

> The majority of these escape codes start with \x1b[

This is actually the 7-bit version for CSI. A less known variation is the 8-bit CSI which is simply \x9b and is the same as \x1b[ combined. So, for example \x9b33m does the same as \x1b[33m .

Unfortunately, there exist a couple of (8-bit) terminals that don't support this version.

I was playing with vttest today. Odd terminal emulator features.

Vttest: https://invisible-island.net/vttest/

Apple's Terminal.app is one of the very few that can pass all tests.

Sadly, it doesn't do SGR 53 overlines. If anyone at Apple is listening, plese, do it.

I haven't tried the escape codes for terminal window movement recently, but I used to use them for window tiling. Terminal.app used them just fine.

I've (edited and) compiled VTTEST to run under Windows. The latest iteration of the Command Prompt does surprisingly well in the tests. Of course the new Windows Terminal does well too.

You can also download the "vt100 torture test" file and "cat" it. http://www.inwap.com/pdp10/vt100test.txt

I really like vttest. I used to use it when I was still working on Console Telnet for Win32 (https://github.com/cout/consoletelnet). It revealed a lot of bugs in how escape sequences were handled that I might never have noticed since they were obscure (yet important) corner cases.

Blatant plug: That time I accidentally discovered my text terminal had a graphics mode.

https://hackensplat.com/falco-t310-unleashed/

Looks like the Tektronix mode some terminals had back then.

https://github.com/rbanffy/fun_with_tektronix

Xterm still has Tektronix emulation. You should see "Show Tek Window" in the middle button menu.

I just suggested it be added to VTE. Not sure the maintainers will like it.

https://gitlab.gnome.org/GNOME/vte/-/issues/328

Hah! I saw your note about programmatically entering Tek mode from xterm. You can do: echo -e "\e[?38h" from the vt100 window to launch the Tek window. Then echo -e "\e\x03" from the Tek window to switch input back to the vt100 window.

Edit: Hrm. I did try it out before posting, and it worked. So not sure why it isn't working for you.

Funny. I tried that while writing it, but it didn't work. These are the character sequences the VT-300 series uses.

> colours are ordered by usefulness

And then I finally realized they’re RGB vectors and additive:

000: black

001: red

010: green

011: yellow

100: blue

101: magenta

110: cyan

111: white

Oh wow, this is amazing!

For proper status lines, use SGR 53 for overlines.

https://twitter.com/0xDEADBEEFCAFE/status/125628536226093465...

You could color code chat messages and your name in Call of Duty 2 in the following way:

^1Red

Everything after the "^1" were red.

The game removed these color codes when it showed your name, but you could do ^^11Red to bypass that (it only removed color codes for one pass)

Coloring names with escape codes sounds like a John Carmack feature

Yup, Quake had something similar that allowed players to have fancy looking names in multiplayer games.

Indeed, this was one of those undocumented but everyone sort of knew features in Quake-engine based games.

Everyone just learned it through asking usually in the multiplayer chat.

CoD 2 used a modified Quake engine :) https://en.wikipedia.org/wiki/IW_(game_engine)

Makes sense since Call of Duty traces it's engine lineage to Quake 3.

Seems to me it's bad practice to combine the text you want to show with the formatting (escape codes) in string literals. Everything is then immediately dependent on specific type of UI technology (terminal).

Eh, depends on context, for bash scripts that might be just fine. You need to have some styling information go along with it anyways, so instead of having some dedicated markup language and then having an output library that converts that to whatever your output uses, you could just declare those ansi escape codes your "markup language" and should you ever need to output it to something else, say a web page, I'm sure there's code around that does the conversion. But I guess to make it more readable you might want to put those commonly used sequences into some variables so you can do

  echo "${esc_red}ERROR${esc_reset}: All messed up"

At least call tput or something so that your script doesn't do anything wacky just because you SSH'd in this time instead of using your customary terminal.

For advanced features, sure. But basic colors are nearly universal.

Until someone runs it in eshell, which is a shell but not a terminal emulator and you have to configure an override just for that special program so that it isn't dumping unreadable garbage into the buffer.

Honestly, since eshell is the only example I can think of, I’d say that it’s more than fair to consider this a flaw in eshell.

Depending on what you are doing you can always parse and translate the escape codes to something else for the systems where they are not supported. One advantage of using in-line escape codes in your string is that you can compose multi-colored text and buffer them in whatever strings your language provides.

Sorry, I don't understand the benefit. Buffer them?

It's clearly harmful to write escape codes in string literals. They are not readable.

The escape code to change any text to red should appear in code exactly once.

This is analogous to HTML inline styles.

Of course. Just view those strings as the output of the formatting backend for the terminal. The article is about terminal escape codes, not software modularization.

Well they wrote: "This tends to result in strings in our code that look a little like this."

That's why I think they have these strings in the code.

> 0 and 7 are less useful for text because one or the other will generally look nearly-unreadable depending on whether the user has a light or a dark background.

This really annoys me about the bright themes used by default in some distros - if you make the default background white instead of black then please also swap the "white" and "black" foreground colors. And make sure all the other ones are readable on the default background for that matter.

I'm a maintainer of chalk. I'm not the first to mention this, but this article is only really parroting the information found on the Wikipedia article for escape sequences, which in turn only really covers xterm-256color terminals.

To be quite honest, I feel as though terminal technology is the most antiquated thing still in wide use today. It's slow, there are so many competing standards and none of them are really that well standardized, and the libraries needed to be truly compatible (terminfo in particular, for which I've written several full parsers and boy what a mess that is) are clunky, outdated and also very slow.

This is a hard problem to solve, however. There are more CSI sequences than just \x1b for example, though that is by far the most common these days. Not treating output as text streams would be an interesting start but that would require rewriting pretty much all software ever to exist.

Meh. I've spent too much time working with escape sequences and thinking about this problem. I am young, but I'll probably die before TTYs do.

Mods: This needs a 2019 in the title (not that escape sequences have been changing rapidly in the recent times :) )

This article was a nice and concise introduction. I wish it had linked to some canonical sources of more comprehensive information though.

Also, the link to the Anki flash cards doesn’t work now.