I don't understand terminals, shells and SSH | Hacker News Reader

I don't understand terminals, shells and SSH

j11g.com

102 pointsjanvdberg59 comments

http://web.archive.org/web/20230114194048/https://j11g.com/2...

O'Reilly's "Termcap and Terminfo" book was very helpful in filling in some of the random holes in my understanding of the tty and it's history.

How useful would you say this information is for the modern programmer?

A lot more than it was 20 years ago?

With multiplatform, CI/CD, cloud, Docker, etc. I spend way more of my time in a terminal than I used to.

I ignore terminfo and just stick to the general VT100 escapes + ANSI. You can expect any decent terminal emulator to deal with them correctly. It isn't worth trying to work with a crusty abstraction mechanism that doesn't offer much benefits today and is largely opaque and hard to manage.

right, but the question was whether or not an understanding of terminfo would help general understanding behind the terminal and tty -- personally I think that still stands, even if you don't insist on using terminfo practically.

But how do you think the information in that book can be turned into practical use for the programmer that spends time in the terminal but doesn't need to write applications that work in (possibly) ancient terminals?

It's useful if you want to write interactive applications from scratch ("curses-like"). Beyond that ... probably not so much.

Read ECMA-48 and DEC VT documentation. That will cover the bulk of what you need to know to operate a TTY style terminal.

Terminfo is basically for smoothing over differences with terminal behaviors that aren't commonly used anymore. It only tells you there's a difference in capability. You still have to implement code to handle them which will never be invoked by anyone but a retro enthusiast plugging in obscure 70's hardware.

It can definitely be useful, but there are also programming jobs/tasks where it's not useful at all. "modern programmer" covers a lot of different cases

It's always useful. Not all programs work best as graphical interfaces. Quite often all that is needed is a text interface, and the easiest way to get one of those is with a terminal of some kind. In today's world, that 'terminal' is most likely to be a terminal-emulator running on a graphical Desktop.

I generally have four or five xterms in use on my average Desktop.

A curses-based text interface is a good substitute for some graphical programs where an addressable cursor is required. It's certainly faster and more light-weight.

Programming for text-based programs also should involve the use of various options given in the terminfo, but most of us are lazy and only cater for one or two main cases. (Hiding behind curses/ncurses as those libraries do the thinking for us.)

If you, or some of the software that you deal with, work in a terminal, you'll have some value of $TERM set. If you use a terminal emulator which isn't one of the classic hits (eg xterm or rxvt), or if you use a terminal mux like screen or tmux, you'll want to have an entry in your terminfo database which describes the terminal accurately, so that simple things like backspace and delete work correctly, or snazzy stuff like 256 color support. So having some kind of general understanding of what terminal capabilities are, how they're expressed in termcap and terminfo, is helpful. In particular, I think having an understanding of the kinds of things that can go wrong in a terminal, for example if you've ever ran cat with a binary file to stdout, is very helpful. It's not the sort of topic you need to spend a lot of time studying, but if you're a UNIX user, I think its a good idea to consider it part of the canon.

Back in 1994 or 1995, a department at my university was giving away some decommissioned terminals. I figured I'd pick one up so that two people could use my PC (running Linux) at the same time.

There were two models available. One model's units all looked cleaner than those of the other, so I picked up one of those.

I bought a cable (RS-232?) and plugged it in. It worked great for command line stuff, but pine (the email client) and pico looked off, with some parts of the screen being drawn over others. I knew nothing about termcap/terminfo. This was before Google (and even before Altavista). I spent hours trying different things (perhaps I got some pointers from helpful folks on a newsgroup?) and managed to improve the situation, but it never worked flawlessly.

Looking back, what I was doing was a lot of random stuff without much understanding. I will probably never know whether what I was trying to do was impossible, or whether I failed just because I didn't know what I was doing and had no systematic approach to editing the termcap file.

For me most surprising thing is that when you do ls and ls|cat it gives you different result. It feels weird that ls knows who's reading it's output.

  if is_a_tty():
    # do…
  else:
    # foo

If only all the younger developers were aware of how to properly develop command line applications.

Linux's Ncurses Howto it's good.

If it wasn’t huge PITA we probably would. Like I try my best to be write well-behaved programs but there’s a cliff of unknown unknowns that is basically bottomless and is known only to people who grew up with the incremental changes.

Stepping through 40 years of organic development by a bunch of different groups whose decisions were based on the limitations of the time is just not gonna happen. But if you don't trying to understand where we ended up will drive you mad. You just pick some closed subset of behaviors / features that exist where your programs run and hope.

I hope someone will create a bridge over the cliff or maybe reinvent a whole landscape without the accumulated cruft over the decades. Till the only the brave will be able to master this space

there’s a cliff of unknown unknowns that is basically bottomless and is known only to people who grew up with the incremental changes.

Don't give this stuff any more reverence than it deserves (which is none). Bob Barton [1] once said "Systems programmers are high priests of a low art."

You don't need to know all the incremental changes, unless the allure of becoming a historian and being able to recount all the developments in the space appeals to you. (Those who do not learn from history are condemned to repeat it.)

There is no mystery except the one you make for yourself. Read books. https://man7.org/tlpi/ is a good place to start. Good luck!

[1] https://en.wikipedia.org/wiki/Robert_S._Barton

Are you also going to do this for the network stack? Graphics? Kernel & syscalls? File systems? DATABASES?? Is there any time left to do work in is there any time left to love your family?

There is also humility in just accepting that my curiosity is greater than my capacity and I'll never know much of anything about anything. I can pick a few things to go deep in but so far it hasn't needed to be terminals and that's not a moral failing.

Being an expert in anything is demanding.

Are we still talking about "command line applications"? Because your comment is exactly how I feel about web development, and somehow all the "younger developers" are more than happy to sit around with seemingly infinitely-complex GUI front-end stacks that only have some indirect relationship to JavaScript.

Tbh I agree with both of you. Things feel fucked all around.

I mean front-end development relies heavily on CLI to compile, optimize, bundle, test stuffs with its gazillion of half baked "make" imitations... every single framework comes with their own CLI tools, and to make everything work together another layer of CLI tools needs to be added on top... which result in slow compilation times, the irony of Javascript in the browser which was designed to be a thin layer of dynamic interactions with HTML documents you only had to refresh the page after editing a script...

It's safe to say that front-end development is now more complex than JEE...

I mean kinda, someone who has been following web development since jQuery and MooTools is going to grok everything way better than someone who is standing today looking backwards, gazing into the labyrinthine nightmare that is the modern JS stack. Either you were around for when each part was introduced and it was a gradual descent into madness where the next step was a natural progression from the previous, or you weren't and now you're in the center of the maze without a map and in your mania you desperately try to invent higher level primitives and metaphors that let you reason about... well anything at all.

But they're all falsehoods that break down eventually and require an elder or time spent with the sacred texts to achieve enlightenment. Repeat the cycle and you get a senior dev in <something: webdev> whose skills are only kinda transferrable to <other thing: CLIs> and you must start your journey over again.

So you have arch mages in webdev who are a acolytes in systems and the only way they'll progress to is write a lot of bad code. I'll get there eventually but how long it takes is dependent on how many opportunities I get to cut my teeth on stuff that requires a more complete understanding.

You don't need to "step through 40 years of organic development" unless you want super-fancy things like an interactive TUI with mouse support or whatnot (and even then, it's not that hard). For most regular CLIs you need a few control codes which are pretty much universal today (setting text attributes, clear line) and that's it.

CLI apps are probably the easiest applications that accept user input to make. Compared to GUI or web apps the amount of knowledge you need is almost trivial.

Decently-sized yet bounded:

- https://en.wikipedia.org/wiki/Advanced_Programming_in_the_Un...

- https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming

I've only read the second cover to cover. Not as much there as you might think, the bulk was laid out in the 70s and 80s when most of an OS could fit in one person's brain.

Not much has been added since, basically extended color and a dozen or so sequences for hypertext links, etc.

I recommend these books. You could use a well-maintained terminal library or two in the meantime.

Every time I pipe the output of `terraform` somewhere I always forgot to append a `-no-color` option.

Every time I also think of just making a wrapper for it, but I never do.

That's kind of what aliases are for.

That's exactly the poor programming I was referring to.

Call isatty()!

The downside of that is that piping to less won't display colours any more either, so then you need "progname -color=always" or some such. Some programs solve that with a built-in "pipe to terminal" feature, but I always find that a bit annoying.

I don't know anything about terraform, but I've written some applications where I intentionally don't automatically change the output (which in my case is usually bold text, I don't use colours much) because I expect no one will ever use the output for scripting . That's not "poor programming", it's a choice because there is no perfect solution that fits 100% of the cases.

I suspect that may probably be the best choice for terraform as well, but I never used it.

But piping to less would display the escape codes, not the colours anyway… which is why this is done.

Imagine grep not working because in between the words that you are searching there is an escape code.

That depends on the settings used for less; AFAIK the default for many distros is to display control characters.

> If only all the younger developers were aware of how to properly develop command line applications.

They are doing just fine. You can't expect them to learn 50 years of technical complexity in a few days and why there are so many different shells and how they behave differently.

edit: replaced debt by complexity

if you want to run with that analogy, command line shells running in terminals are assets, not debt. Not knowing how to use them is poverty, lacking having assets.

you can't just take the word "debt" to mean bad, then just throw it around for things you don't like, especially applied to things that take effort to learn, effort you have not and don't want to expend.

There is no "50 years of technical debt". If anything, a significant amount of complexity has been removed compared to the 70s and 80s when everyone and their mother was making wildly incompatible terminals (actual terminals that is, not software terminal emulators).

You don't really need to know anything about shells to write a CLI app, other than basic shell conventions (- for flags, things like that). You don't even need to know all that much about terminals, just some bare basics if you want to do colours or whatnot. All of this can be summarized in a page or two.

If you want to write a ncurses-like library: that's a different story. But then it's no longer a CLI app.

The parent poster was likey refering to the fact that good apps should suppress colors when output is not a tty, so that all the text-processing tools work properly.

This works identically in all the shells (and even without a shell), and there is no "50 years years of techinical debt" (unless you mean the idea itself of in-band control codes)

If only programming were that simple...

  if(operation_failed)
    make_operation_succeed();

Actual Python 3 code, in case you're not convinced

    import os
    if os.isatty(1):
        print("stdout is a tty")
    else:
        print("stdout is not a tty")

It really is that simple. Substitute the 1 for 0 when checking if stdin is a tty, and 2 if checking stderr

I've actually seen code that is basically:

    if (! try_thing())
       if (! try_thing_again())
           if (! try_thing_a_third_time())
               give_up();

... where the thing is something that should basically always work, and if it doesn't there's no good reason to expect that trying again will have a different result. The programmer who wrote this thought it was defensive coding and a good practice.

The function is literally called isatty, so it's not that far off

That's very different. This is functionally checking a flag[1], and acting accordingly. There's nothing to fail or make succeed.

[1] There's an ioctl involved, and it can fail for other reasons than "fd is not terminal", however all the reason for the operation to fail are good reasons to not proceed with terminal specific behavior.

Many programs do this in fact! And it's great because on a terminal you might want colors or abbreviated file names for readability.

isatty(3) is a common way to do this. However, it took typical CLIs a while to actually do something useful with it. Turning off colours is one thing you might want to do. For me, git with its automatic pager support has done the first really useful showcase of that.

Funny thing is that I thought I understood it, generally, but I just made a little live coding video as I was trying to discover how to build a shell in Go. Although I was successful, I’m still not entirely sure how this works. I figured out how to build and run a command for SSH users, which was my goal, but that’s not quite a shell.

I’m bookmarking this to come back and read through some of these resources.

If you’re interested in my own process, here are the video’s I recorded as I worked through this.

Part 1 - Building the command https://youtu.be/s4ggSklJKic

Part 2 - Installing it on a Raspberry Pi https://youtu.be/G968wOZGFGs

https://devblogs.microsoft.com/commandline/windows-command-l...

I used to log in via modems and terminal servers of various kinds, so I found this to be an interesting reminiscence of olden times... And yes, we take a lot for granted these days - even the backspace key :)

This should be one of those abstractions that just work but from time to time it pops up with arrows suddenly not working while jdb or other places. Somewhere some escape codes are not right

I sometimes ask candidates to talk about these exact things during interviews. Describe what happens when you type a command into a shell. Then how about if it's in an ssh session. It's a good way to dive into all kinds of stuff from syscalls to networking. I look for an overall understanding and curiosity about how stuff works.

These are SRE type positions where strong systems knowledge and debugging skills are a requirement or at least a big plus.

Makes sense. It's something where even if you don't know the true details, you can engage in intelligent speculation about how it could be implemented.

That's fine to ask but a non-answer gives you almost no information other than perhaps that the candidate never bothered to learn in great detail exactly how it all works because it's never been important enough to spend time on.

If you're looking for strong diagnostic skills, then ask those questions geared to give you direct information as opposed to trying to weed it out from the answer to a vague high-level process description question. If the candidate has no idea why you're asking then any answer at all should be satisfactory.

Weeding out how to approach well when the ask isn't clear or you don't have full understanding of the problem space is a great skill to select for and doesn't require any technical knowledge at all about what the question is asking. A true non-answer (not just a lack of deep technical understanding on what really happens) should hint to further investigate how strong a candidate is in that area not hint they aren't technically savvy.

It's hard for me to imagine any strong Unix systems engineer that knows nothing about that topic. The whole tty subsystem is one of the core parts of *nix systems, and basic knowledge of it is required for many complex debugging questions.

And the open ended question is great because it allows candidate to showcase what they know. Have no idea how PTYs are allocated by kernel? No problem, you can talk how shells work.

While I've been using Unix/Linux shells for seemingly ever, remotely at first with telnet, then with SSH, I have a fairly consumer point of view towards the entire stack: I know their components well enough to be able to use them in my every day work, and to debug them with the few problems I encounter. The nitty-gritty doesn't concern me, and I frankly don't care much. I rather learn other stuff in more depth. That said, I do muchly appreciate that I can afford this point of view, because everything's stable and user friendly enough that I don't have to care.

Disclosure: I'm not an SRE

Also as someone who was always multi-platform I didn't know the nitty-gritty until recently myself, and honestly knowing it is just for fun for most folks. TTY/terminal-centric problems just don't happen any longer, and if they did you can boot into recovery mode or "nuke the OS from orbit" with a live drive.

Things that do happen with some regularity like disk corruption should probably get focused on first.

Just discovered what seems some kind of race between vimrc variables and vim syntax highlighting only happening... via ssh?!

well it is gone, my bad.

Me too, just don't be afraid.

Reply on news.ycombinator.com