Character by character TTY input in Unix, then and now | Hacker News Reader

Character by character TTY input in Unix, then and now

utcc.utoronto.ca

72 pointsg0xA52A2A31 comments

Would you agree that the whole TTY subsystem makes very little sense in this day and age? Why not let programs operate in raw mode by default and provide user-level library calls for input line buffering?

It's important for job control. Like Ctrl+c or Ctrl+z.

Arguably the shell or terminal could handle those.

I don't know. It is one of the recurring questions in software development (or politics...) what should be centralised and what should be distributed. Job control might be one thing that we need a good default for anyway, so why not implement it in a central place?

Why not do that screen/tmux style, where shell takes input and passes it to foreground program, unless it's one of those special key sequences?

That's not what happens with screen or tmux (or xterm, or sshd on the remote side of an ssh connection...). These allocate a pseudo terminal (PTY; which is a structure in the kernel) and connect to the master end. Using that they emulate a terminal, which has a virtual keyboard and a virtual screen. A shell (or another program) is started, with its input and output(s) connected to the slave end.

The virtual keyboard might be X11 key events in the case of xterm, or a network stream in the case of sshd, or simply the stdin downstream from another terminal in the case of tmux. That keyboard character stream is fed into the write fd of the master end of the PTY.

The virtual display is implemented by processing data read from the read fd of the master end of the PTY. That might involve drawing to an X11 window in the case of xterm, or sending back data over the network in the case of sshd, or writing to stdout connected to another terminal in the case of tmux.

The important part is this: The job control is still done in the kernel as part of the PTY implementation. That implementation is where for example a Ctrl+c (0x03 byte) is converted to a SIGINT which is then sent to the slave program.

Raw mode by itself is essentially useless for almost all programs; in order to do anything with it, you already need an entire surrounding collection of code to handle editing (even basic things like backspacing) and so on. But line-oriented input remains useful on its own without additional user-level code, again precisely because of the editing support the kernel already provides for you. Since you already need user level code to make any real use of raw mode, the user level code can also set things into (and out of) raw mode as you need it, so line mode remains a sensible default.

If anything things should move the other way, with the kernel providing more readline-style line editing in 'line mode' so that programs need to implement less of it themselves. But this would provoke all sorts of arguments over what line editing the kernel should support and so on, and then someday someone will suggest that the kernel should let you implement your own in-kernel line editing using eBPF (or WASM, if one leans that way), so people could push their preferred choice into the kernel when they log in or start up a shell on a new pty.

(I'm the author of the linked-to entry.)

You are Kenneth Almquist, and I claim my five pounds.

* https://groups.google.com/forum/#!msg/comp.sources.unix/Vejy...

Funny you should mention that:

  ; ls -l <archives path elided>
  -rwxr-xr-x. 1 cks man 65536 Dec  4  1991 ./bin/bin.ultrix-mips/atty

I see that GIDs around here have drifted a bit over the years.

The fact that the kernel TTY driver is responsible for handling erase in cooked mode leads to a wrinkle: in order to do that correctly it needs to know if you're using a multibyte character set, and if so, which one (because a single erase must erase an entire multibyte codepoint).

Linux uses the (non-POSIX) IUTF8 termios(2) flag to indicate a terminal is using UTF8. If this is set on a terminal which isn't UTF-8, you'll see some odd results - for example, in a terminal using the single byte character set ISO-8859-1 but with IUTF8 set and in cooked mode, if you type:

  Ã¡Ã¡

and then backspace once and hit enter, the process reading cooked mode will recieve "Ã¡\n" - the erase has eaten two ISO-8859-1 codepoints because it thought that they were a single UTF-8 codepoint.

I've never used a non-UTF-8 multibyte terminal (like BIG5 or SHIFT-JIS) but I don't think erase would work correctly in cooked mode on such terminals.

This is incredibly timely. I've been trying to figure out if it is possible to create a universal shim that would always give me readline-like behaviour while supplying input to an ongoing program.

Something like this:

./shim.sh | ./some_horrible_cli.rb

That way I could always rely on my linux shortcuts. Ie, no "^A^K" when I try to kill a line; it would just kill the line.

Is this possible?

Good news! This already exists. It's called 'rlwrap' and it's awesome.

Thank you! :)

Correct me if I'm wrong, but I don't think that protected mode existed in old Unix systems and so was not delineated between user space and kernel space as described.

Protected mode and the separation of kernel and user space was only available since 80286 processors starting in 1982.

Refs:

https://en.wikipedia.org/wiki/User_space

https://en.wikipedia.org/wiki/Protected_mode

Virtual memory systems and protected memory certainly did exist before the 286, and Unix supported them well before 1982. Hell, the most popular CPU architecture for production Unix systems -- VAX -- was 32-bit and dates from the mid-1970s.

Sorry, but you are wrong.

Have a look at the architectures of the PDP-11, VAX, 68000 and many other CPUs.

By the way, an interesting story:

When Sun (?) wanted to implement their Unix with paging on the 68000 they had a problem: If you access a memory page that has been swapped out to disk, the OS has to (1) load the page from disk to RAM and (2) let the CPU repeat the instruction that caused the memory access. But the last step could not be properly done on the 68000 because it did not store its internal state completely when an (address) error happened (this was fixed in the 68010).

Their solution: Run two 68000 in parallel on the same code, one of them delayed by one instruction. When the first CPU triggers the page fault, the system can stop the second CPU before it reaches the instruction that caused the fault.

That's astounding. 68ks were not cheap. I assume the system then stalls for a couple of instructions to switch back to the first chip after the page fault completes? I have to assume they reached that solution after exhausting every other option.

On the other hand, this could also be used as a form of error checking. If CPU2 ever returns something different you know there was an error somewhere in the system and you can hard stop to prevent further data corruption.

> I have to assume they reached that solution after exhausting every other option

I assume that the 68000 was so powerful for its price (probably much cheaper than the existing mainframe CPUs) that it made this a viable solution. Or maybe the company had promised their customers a 68k-based Unix system (with the 68451 MMU) and the 68010 was delayed or too expensive and they had to find a quick solution? (I have no idea)

Btw, I was probably wrong about Sun being the company behind this. Apollo and MassComp have been mentioned in mailing lists.

IIRC they cost around $350 each back in the 80s. That's roughly $1000 in today's dollars.

A lot of companies wanted to use them instead of the Mickey Mouse Intel chips of the day, but the price point was too severe.

Speaking of which: http://www.os2museum.com/wp/the-nearly-ultimate-386-board/

The article describes a motherboard with both a soldered on 386 CPU, and a separate socket for another one. This was not a multiprocessor board, to make use of the socket you were supposed to disable the onboard CPU using a jumper.

But if you plugged in a similar enough CPU without setting the jumper, it appears that both CPUs ran at the same time. And since they had the same clock and they should have no difference in behavior, the system, at least as tested, ran fine.

Hah, that is amazing and baffling at the same time.

Elsewhere, you unfortunately sometimes have to plain emulate the instructions. VM86 mode on x86 is notorious for that; not so much when accessing memory but for all the vast privileged instructions that can trap into the monitor (e.g. CLI, IN, OUT, POPF...). VM86 Extensions ameliorated it only somewhat.

But VM86 also does not have much relevance anymore. It was a mode for running 16bit real mode software, and 64bit x86 CPUs don't support it anymore when in long mode.

I'm having a hard time understanding how this would actually work in practice.

CPUs act on data coming from eg the disk, memory, the network, etc. The results of instructions influences what the CPU will do next, especially so in the case of self-modifying code.

So, with a system that has a "chaser" CPU following a master CPU one instruction behind... how do you also impose a one-instruction delay on the CPU's view of the real world?!

Unless this is done, the chaser CPU is going to see real-world data and I/O that is out of sync with the instruction stream, isn't it?

I'm quite sure this problem was very elegantly solved, and I'm very curious+interested to find out what that solution was.

There is a world outside of Intel, especially back in the 80's and 90's...

And especially back in the '60s and '70s, when Unix and its predecessors were originally developed.

wrong chip.

https://en.wikipedia.org/wiki/VAX#Virtual_memory_map

(although paging was in pdp-11 unix too, and neither was the 1st to have memory levels)