SSH productivity tips | Hacker News Reader

SSH productivity tips

blogs.perl.org

330 pointsbigiain65 comments

You'd be far better off learning screen or tmux than by using shared connections. Knowledge of that will improve your offline options as well, will transfer to any ssh client (including Putty), and IMO is far more useful than having two separate terminal/ssh windows.

Agreed. I actually did a Cmd+F "tmux" and "screen" after skimming the article to make sure I didn't miss it somehow.

A terminal multiplexer is an absolute must if your connection is likely to be interrupted. Closing the laptop, moving to a different room and picking up right where you left off is great.

If ssh servers and clients didn't use their own "keepalive" messages, you'd be able to do that anyway. TCP is supposed to be able to pick up right where it left off, so your ssh connections shouldn't die if you lose connection for a few minutes.

man ssh_config:

TCPKeepAlive

Specifies whether the system should send TCP keepalive messages to the other side. If they are sent, death of the connection or crash of one of the machines will be properly noticed. However, this means that connections will die if the route is down temporarily, and some people find it annoying.

Turning off TCPKeepAlive also could mean that stale connections are killed. We have a router that kills connections that have been stale > 2 hours. I run my screen session on a single server and ssh to all other servers within that screen session. Without TCPKeepAlive, I would lose most of my session overnight (or after a string of meetings).

If you lose some packets for a few minutes, it shouldn't die,but there are a zillion real world situations where it can and WILL die....... and it's not an SSH problem, screen was useful for the same reason long before ssh.

I don't know - I like having the control master around for when I've done something like startup an interactive process, then realize that I also want to pop onto the machine to look in some file. I don't always start screen when I login somewhere, and I don't particularly like screen well enough to make it my default first action when I log in somewhere. I also tend to prefer to be able to see all my logins simultaneously, rather than having them be screen windowed or tabbed.

I thought it was a nice overview of some things I use a lot (control master, and nicknames in .config) and included some things I didn't know, like the control persistence and some of the nickname wildcarding. It shouldn't be dismissed just because it isn't about screen/tmux.

If it's a long-running process, you might want to consider screen if only for the protection against connection issues. Otherwise, you could potentially combine the control master with `screen -x`, or just use a split window in screen/tmux, to allow viewing multiple windows at once.

I tend to edit files locally and rsync (the changes) to the server, running rsync over ssh. I do this fairly often (edit - save - rsync - check), and avoiding time to set up a new ssh connection is a huge speed boost to my use case (since the delta is usually quite small).

Even if I don't need to be logged in to the server at all, I still do it so it keeps the master connection open, and then speed up the rsyncs.

Have you considered doing something like pushing with git?

Every push with git is going to create a new ssh connection. Using a shared connection is way faster.

This would suffer from the same problem.

But doing git push all is out of the question for me, because I'm rsyncing while developing, so I'd need to constantly commit and push tiny nonsensical commits to the server, and then do a massive rebase afterwards. In effect, it's more work and gains me nothing.

I hope this is some sort of staging or testing server?

The idea of repeatedly writing code and rsyncing it up to a production server while adding new features/whatever seems incredibly error prone to me. And that's not even to touch on whether or not there's automated testing (in the form of unit/integration/whatever tests in the code).

Yeah, it's a development server. I've got a cheap VPS (€4/mo with 512MB of memory) and I usually run the code I'm developing there.

My reasons: I don't have to pollute my laptop with various (sometimes mutually exclusive) servers - virtualenv &co are great, but can only go so far, and VMs are much harder on the battery; trivial for other people (colleagues, clients) to see WIP, even if my laptop is suspended.

Which provider is that ?

alvotech.de

Awesome. Will try it. Cheaper than linode. Curious how they perform.

You should give Emacs and Tramp some consideration. Failing that, Fuse works well.

shared connections with ControlMaster etc are really handy for things like scp or sshfs.

Using TRAMP for emacs to tab-complete remote filenames is really nice, and much faster if you have a control connection already established.

I've not come across the ControlPersist option before - it's something I'm going to have a poke with to see if it can replace my scripts that fire off a bunch of common host control connections in the background, and use autossh to resume them when I get a net connection or wake the laptop from sleep.

I use ControlPersist, but it doesn't seem to work well with mercurial. If hg makes the first ssh connection, hg won't exit until the master closes. ssh in a terminal then running hg doesn't have the same problem. The persistent connection child isn't detached enough apparently and hg waits for it. I'd be careful about testing auto connection sharing to make sure things don't get hung.

Yeah, the idea is that you ensure that the "master" channel; the first one, is always backgrounded (and connected), otherwise I intercept the ssh command and have it spawn a new master first.

The biggest downside to ControlMaster in general is when you accidentally close (well, kill) the master session, and everything else dies along with it. The usual behaviour is to close that channel on EOF, but wait until all other muxed connections are closed before actually exiting. If you forget it's the master and Ctrl-C it, bye bye sessions.

My backgrounding stuff is currently a horrible mess of Perl scripts, and a bit flakey when it comes to hostname/option parsing, but I might give it a tidy and stick it up on github at some point.

ControlMaster auto and ControlPersist are supposed to do the backgrounding automatically. I think it's just not "far enough" in the background.

You can turn it off when using Mercurial by putting something like this into your .hgrc file:

    [ui]
    ssh = ssh -o ControlMaster=no -o ControlPersist=no

The point, though, is that I want it on (and working) because it's fast. hg is responsible for the majority of my ssh connections.

My problem with nested terminal emulators is that they reimplement the scroll buffer (by necessity). This in turn breaks the native buffer scrolling keyboard shortcuts of the outermost terminal. For some servers, this is worth the pain; for others, it's not.

So use a better terminal emulator. Rxvt-unicode allows you to pass mouse drags and scrolls directly to the terminal by holding down shift, and since it supports perl extensions, there are many ways to scroll back and yank from the terminal's buffer.

Tmux is great even if you're not using it as a process manager; it dramatically increased my productivity at the terminal (which is most of my day)

Emulating as rxvt doesn't seem to have the desired effect on iTerm2 on OSX. Is there another solution? Entering copy-mode on tmux/screen isn't ideal.

@TREYisRAD

No, you'll want to actually use rxvt-unicode [1] on X11 on your Mac. The X implementation is annoying at times on OS X, but using a great terminal is worth it if you spend all your time in one. iTerm2 is nice, and Lion's new Terminal.app isn't bad, but programs like vim and tmux are designed with real xterm-compatible terminals in mind [2].

[1]: http://dist.schmorp.de/rxvt-unicode/

[2]: For instance, you can enter copy-mode in tmux 1.5 with a mouse scroll if you are using xterm or rxvt-unicode

The developers of iTerm2 and tmux are working on better integration: http://code.google.com/p/iterm2/wiki/TmuxIntegration

I'd love to, but I'm not a big fan of having to run X. Rxvt native Win32 build (as shipped with Cygwin) is my standard emulator, nice and light; and the way it integrates with the Windows clipboard makes it work even better than on X (i.e. the selection buffer - I hate that) for my usage style.

in most environements you can merge the selection buffer with the clipboard (or dismiss the selection buffer if you prefer that)

.screenrc

defscrollback 30000 term xterm-color termcapinfo xterm ti@:te@ # scroll wheel action!

escape ^Tt # Instead of Control-a, make the escape/command character be Control-b

Any good resources for learning screen? So far I've only mastered:

"Ctrl-a d" and "screen -r"

I'd love to hear how you use screen, for example. Do you use multiple sessions or windows? Split screen?

You're missing out. After (or maybe alongside) those two keys, I find the most useful ones are:

  Ctrl-a c (to create a new terminal)
  Ctrl-a 1 (to go to the first terminal)
  Ctrl-a 2 (second, etc)

If you're an emacs user, you might also want to change the command key away from something as essential as the default bind for beginning-of-line. I myself use Ctrl-T. In .screenrc, that would be:

  escape ^Tt

The screen manpage is also very good.

Add to that list C-a " to view the list of screens. C-a A can be used to assign names to those screens. C-a [0-9] can be used to navigate directly to a specific screen. C-a C-a will swap the current and previous screen. C-a S splits the screen vertically C-a <tab> navigates to the next visible screen (there's no back-navigation). C-a q collapses all splits leaving the current screen visible. C-a d detaches from the screen session.

There are some other features, but with these you're using much of the power of screen. Takes a few days to get used to, but it's one of those "how did I ever live without this" tools.

I personally use screen in a relatively "simple" way. I don't use splits, but I do occasionally use `screen -x` to have multiple views of the same screen session.

My biggest use of screen is for running a persistent Irssi session on my server, and connecting to that from whatever terminal or mobile device I happen to be sitting in front of. In that case, I have just a couple active windows, but only one that I use 95% of the time.

I have a similar persistent screen session on my work machine for running Irssi's SILC client to chat with co-workers.

I also use screen for my primary workflow, because a lot of my development takes place on servers. I'll start a screen session on each server, and create a new window for each type of task I'm performing. Eg, I have a window each for `htop`, mongodb console, vim, git, and for running the program I'm working on, plus a few others for one-off tasks like installing a new package or editing a system configuration file.

Some tips I have include:

- Set up a nice .screenrc file, and make yourself a customized "caption" line, which will give you a status bar of sorts that will list all your open windows. Mine is `caption always "%{= dd}%{+b ky}%{+ .b} $LOGNAME@%H %{-} %{.y}%-w%50>%{+ Kg} %n %t %{-}%+w%<%{-}"`

- Similary, either get in the habit of naming your windows, or set up your shell to properly report commands to screen so it can automatically name them based on the command you've run. My zshell config does this for both screen and gnome-terminal: https://github.com/jreese/oh-my-zsh/blob/jreese/custom/scree...

- Ctrl-a, Ctrl-a is a great combination for "alt-tabbing" between windows

- I set my least-used window to #0, as it's the "hardest" number to hit after ctrl-a, and I put my most used windows between #1 and #5 for easy one-handed switching.

- Ctrl-a Escape is the easiest way to get into copy mode to view your scrollback buffer. Ctrl-a [ is equivalent, but IMO using escape instead allows me to start moving my right hand to the cursor/paging keys while I'm still using my left hand to enter copy mode.

https://wiki.archlinux.org/index.php/Screen and http://en.gentoo-wiki.com/wiki/Screen are my go-to pages.

  screen -x

Attaches a currently running session without detaching it from where ever it's currently attached to. (Note: this can cause some issues if the currently attached session has a larger/smaller terminal).

I've also mapped Ctrl-a to ` instead. This makes flipping between the most recent 'windows' as easy as `q.

  # Use backtick as the escape character. Use F11 and F12 to switch between ` and
  # C-o as the escape sequence. This is useful when pasting text to the terminal
  # that may contain backticks in the text.
  #
  # SOURCE http://superuser.com/questions/74492/whats-the-best-prefix-escape-sequence-for-screen-or-tmux
  escape \140\140
  bindkey -d -k F1 escape ^O^O        # bound to F11
  bindkey -d -k F2 escape \140\140    # bound to F12

F11 swaps the escape sequence to C-o (useful for pasting things that might have a ` into the terminal) F12 swaps the escape back to `

Also:

  [escape] n  -- cycle to the next window in the list
  [escape] p  -- cycle to the prev window in the list
  [escape] "  -- see the window list

Turn off the startup message:

  startup_message off

* Note: 'escape' doesn't refer to the Escape key, but to the key sequence that 'escapes' all screen key bindings (default: C-a).

  info screen

Not sure why this was downvoted. "info screen" at the command line is exactly how I learned about all of screen's features.

The choice of C-a as the Magic Screen Key Combo is unfortunate.

I start my screen sessions thus: screen -e'^\\\'. This makes C-\ the Magic Screen Key Combo. C-\ has other meanings in some contexts, but in practice I've never used it. On the other hand, I use C-a lots, in Emacs and in shells.

Also, if you plan to have more than one screen session, you can name them with -S <argument>. So: screen -e'^\\\' -S blahblah.

You can reattach later: screen -Dr bla. If there's no ambiguity, you don't need to type out the full name.

Connection sharing and terminal multiplexers aren't mutually exclusive. Let's say that in order to get to 4-5 hosts (which you run screen on) you have to jump through a certain SSH bastion host which you specify using the ProxyCommand directive. You're going to want connection sharing on that bastion host, to make subsequent connections to hosts beyond much faster. If you have to jump through two bastion hosts to get where you have to go (yes I actually have to do this) connection sharing is a must.

Even without a bastion host, connection sharing allows for faster scp from your home box to the target host you're running screen on.

No, screen/tmux does not solve the same issues shared connection do.

In fact, they've little to do which each other. Screens allow you to keep your session running and reattach to it. Control (shared connection) allow you to keep the connection running instead of opening multiple ones.

This is much much faster for example if you're using SCP, or have scripts opening different tunnels, and so on.

I think the point is the "normal use case for screen", which tends to be, for me: keep alive, and sometimes multiple sessions in one window.

Here's a minor one - handy, and I find most folks don't know it. To escape a frozen SSH connection:

<return>~.

<return>~? gives you the help screen that lists all the SSH escape sequences.

I was surprised that this didn't mention SSH compression, which makes a noticeable difference on a slow connection:

ssh -C me@myhost.com

you can also specify a compression algorithm

(man ssh) -C Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11 and TCP connections). The compression algorithm is the same used by gzip(1), and the “level” can be controlled by the CompressionLevel option for protocol version 1. Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. The default value can be set on a host-by-host basis in the configuration files; see the Compression option.

> If connecting to a server seems to sit there for a few seconds not doing anything, try adding this line to your config: > GSSAPIAuthentication no > And if that works, ask the server’s sys-admin to disable it in the server config, for the benefit of all users ‒ exactly the same line as above, but in /etc/ssh/sshd_config.

It would be nice if they described what this actually did - I'm hesitant to run options off, especially ones related to authentication.

It turns GSSAPI [1] off, which is typically used with Kerberos. Unless you know you're using it (and in most cases you will know if you are), it's safe to turn it off.

[1] http://en.wikipedia.org/wiki/Generic_Security_Services_Appli...

And GSI-Openssh too: http://grid.ncsa.illinois.edu/ssh/

The other really common case with delayed connections is that the sshd is waiting on a timeout while it tries to look up the rdns for your connecting IP.

    UseDNS no # disable rdns lookups in sshd logs. See man sshd_config

There's a possible security benefit to having it enabled, in that it'll try to resolve your rdns, and then confirm your forward dns matches it, but I'm not sure how much of a benefit that is, in practice.

> "SSHFS works on Linux and OSX. I haven’t found anything equivalent for Windows users."

There's Dokan SSHFS, (Dokan is a similar endeavor than Fuse FS, only for Windows). For a commercial product, there's apparently ExpanDrive.

Just a warning with shared sessions. If you use in shell vi be aware that killing your first connection will kill that other shell you are working in.

This. I did this so often I stopped using shared sessions.

Going a bit OT here, but I'm pretty sure there are a fair few Mac users here: has anyone noticed the ssh-agent/OSX Keychain integration behaving differently in Lion compared to previous versions?

In the past, I'm sure that waking the laptop from sleep, or unlocking the screensaver also unlocked the login keychain, and with it updated ssh-agent with my priv key, but recently I've been having to manually unlock it, which is getting to be quite a pain.

(ssh-agent remains running the whole time, ssh-add -l shows the identity when the login chain is unlocked, but is empty otherwise)

Yes, I've noticed something similar, although I do not let it store my ssh password on the login keychain. First time I ssh after a reboot, it asks me, I enter it.

Pre-Lion, it was then good until the next reboot. In Lion, it occasionally asks me to re-enter it. As you noticed, this seems to correlate with coming back from sleep or screen lock. It doesn't seem to happen every time, though, so I'm not sure what is going on.

According to ps, it is using the -l option to ssh-agent. The man page and the usage message of ssh-agent do not admit the existence of such an option. Perhaps that has something to do with it.

Your comment about the mysterious `-l` option got me interested and I did some investigation. It looks like a custom option added by Apple to indicate it was started by `launchd`. Here's a link to the relevant patch on their opensource site:

http://www.opensource.apple.com/source/OpenSSH/OpenSSH-95.1....

Nice trick using a proxied netcat command to forward ssh connections through a server. Was not aware of that, and I spend a lot of time running through a front ssh server, thanks!

Not sure I understand the point of it, it's trivial to do that without netcat.

  ssh -t server1 ssh server2

Will bounce you right to the second server. I'll generally alias this as the second servers name.

with netcat you don't have to use agent forwarding. its better in many cases because you have use different keys for the jump host and the destination host for example. trusting the jump host by using the same key is not very good security wise, as compromising the jumphost would mean compromising all hosts on the network for all users. using agent forwarding is equally bad of course.

nc solves that.

Likwise if you use this for scp, it works with nc, but it wont with a double ssh.

Ah, interesting, thanks.

For interactive multiple file transfers using SSH as an underlying transport, both lftp and mc are pretty darned good. 'lftp fish://user@hostname/path' sets up the basic connection. From there, it's like using a full-screen ftp client (lftp also supports ftp, sftp, and multiple other protocols).

mc can access remote filesystems and do complex file and directory management with ease.

Of course, there's always rsync as well.

I like to keep my server's filesystem mounted locally (via nautilus/gvfs+bookmarks) and edit files using a local application. I also keep a few terminals open as well to run commands, restart things, commit changes, etc.

Fast, easy to use, and nothing special has to be configured at the server besides installing sshd.

before you enable agent forwarding (and btw it's "ForwardAgent yes" not "AgentForwarding yes") you may want to enable confirmation in ssh-agent:

http://jcs.org/macssh

Hi guys, posted a related poll at http://news.ycombinator.com/item?id=2895035 - please vote! Thanks.

Reply on news.ycombinator.com