SSH Examples, Tips and Tunnels | Hacker News Reader

SSH Examples, Tips and Tunnels

hackertarget.com

376 pointsthewanderer199970 comments

If you liked this article, PLEASE, PLEASE just take 10 minutes to look at the manpages for ssh(1) and ssh_config(5). SSH options are nothing we need to research about, we humans made it and we wrote documentation for it. The manpages are also exhaustive, definitive, and valid for the exact version you have installed.

Please, instead of making more 'how to use SSH'-articles popular, read the manpages yourself.

Reading the datasheets and documentation of the stuff you work with is an important characteristic of an capable engineer.

Do you want to get rid of how-tos? Both references and how-tos are needed. They serve different purposes.

> Reading the datasheets and documentation of the stuff you work with is an important characteristic of an capable engineer.

I completely agree with this statement, but that doesn't nullify the value of "how to use x" articles. Man-pages are great because:

- they're (mostly) ubiquitous

- they're plaintext, meaning their format is more robust and accessible (e.g. I can read man over SSH without ever encountering issues)

and these are great pros, but in certain relevant aspects they're grossly inferior to articles like this:

- man-pages must be exhaustive, and thus don't have the luxury of focusing on the most common use-cases up front. These are often relegated to the "Examples" section only, or omitted altogether

- screenshots. Even for terminal apps, screenshots are incredibly helpful, and in this particular article they're also used to demonstrate the use of applications your SSH connection will interact with (e.g. browser proxy/network settings)

- (related to the above) contextual information that would otherwise be off-topic/inappropriate in a man-page. How-to articles can cover multiple apps that interact with eachother as one topic; i.e. a how-to article can focus on a use case rather than a single app.

While your point is valid it applies to a single tool. Often the combination of multiple tools and Unix knowledge is what provides a simple solution to common problems. Not everybody has the broad knowledge to come up with these themselves so tutorial articles are a valid and helpful complement to man pages in my point of view.

Man pages and other documentation are usually reference oriented. That means they are complete, verbose, detail-oriented and structured around functionality and options.

When learning, however, we often need goal directed material and examples. For multi-purpose tools - and most tools have more than one application, especially for higher level goals - that means there can be many different introductions for the same tool, depending on the learner's goal.

The best tools are compositional. Often it's hard to imagine all the ways a tool can be composed by looking at it in isolation. You need some examples to get the imagination working.

My usage of manpages would increase by orders of magnitude if I knew there were examples of actual working commands in the first page. Most of the time I am not looking for an obscure parameter, but my brain just has failed to take a note of the basic syntax of the command. So why not start ssh manpage with two simple examples, how to connect to a host with username and ssh key:

ssh user@example.com

ssh -i ~/.ssh/id_rsa user@example.com

indeed. I've always thought this, would be nice to see examples that might cover my need. Then read what the switches/options do.

So you're just going to run some command without knowing what it does, just because it looks right?

That is the reason why I like the Golang documentation; it consists of exhaustive comments, (runnable) examples, explanatory code snippets and links to the corresponding source.

Works as a reference but also for explorative learning.

There are a few projects out there trying to do this as community driven efforts or by extracting examples automatically from the existing man pages (or a mix of the two).

https://tldr.sh/ and http://bropages.org/ spring to mind, and I know there are others that I'm not immediately remembering ATM, both of which have web interfaces or can be installed as local command-line tools.

This is how every man page starts, with a "SYNOPSIS" of invocation. For example the ssh man page starts:

    ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b bind_address] [-c cipher_spec] ...

If there are several incompatible ways to invoke an operation several cases are given, e.g.

    netstat [-AaLlnW] [-f address_family | -p protocol]
    netstat [-gilns] [-v] [-f address_family] [-I interface]
    ...

Typically there are examples in the EXAMPLES or USAGE section which you can jump straight to via /^U or /^E

In the case of ssh there are several such sections so just jump to them via /^[^ ]

      > ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b bind_address] [-c cipher_spec] ...

That's exactly the kind of "SYNOPSIS" would make me want to stomp on puppies when looking at a man page because I forgot some basic usage.

But actually, thanks for the key-combo tip on jumping straight to examples.

I find many manpages to be pretty obtuse. They could do a lot to help those of us who need a bit more hand holding.

I like cheat.sh. You can curl it directly from your terminal and it offers the quick examples I often need to get back to work.

Here is their ssh page: http://cheat.sh/ssh

Also I am a huge fan of Matlab’s Documentation, at least up to the more recent versions that became less usable. They give a clear description of the function, list syntax options, give several examples and provide a see also section with similar or related functions (greatly helping discoverability). They usually include academic references at the end.

https://uk.mathworks.com/help/matlab/ref/atand.html?s_tid=do...

Whereas it works great for me so that when I can’t remember the precise syntax I can just glance at the top of the man page; if I want anspecific option I can search for it.

Info format files are intended to be more comprehensive and tutorial but they never really made the jump to Unix.

Sorry, in my Ubuntu 18.04 "man ssh" produces output that does not have either EXAMPLES or USAGE. /^E takes me to ENVIRONMENT instrad of EXAMPLES.

Yes but as I mentioned in my comment for ssh there are several sections that each explain a different use case which you can jump to via /^[^ ]

Sorry, I am confused. In your comment you said that in case of ssh there are several EXAMPLES or USAGES sections. Well, in my case I find exactly zero instances regardless what kind of key combo from your comment I use.

If you instead meant that e.g. TCP FORWARDING or X11 FORWARDING are such sections that explain use cases, well, in that case I still fail to find a section that would describe simple use case of just connecting to a host.

> My usage of manpages would increase by orders of magnitude if I knew there were examples of actual working commands in the first page.

Try adding ManKier as a custom search provider for Firefox or Chrome, it is online man pages beginning with TL;DR examples at the top of each web page, followed by the traditional man page entry below. Here is a fuller explanation with links to instructions for adding it to your browser: https://www.mankier.com/about

I use it almost exclusively now, and only refer to my distro's man page in a terminal if I need to check on something that isn't standardised (such as sed -i.bak or something like that).

Custom browser search is pretty handy, try adding a dictionary and map provider too, or 'caniuse' if you do front-end work.

I don't really want to read dictionary cover to cover to learn english.

Sometimes all you need is a cheat sheet. Deep understanding of the tools is valuable, but not always have good ROI if it is not in your core task.

Especially for standard/common/trivial tasks.

It can bite your ass of course, but it is rare - and if you do something more exotic - take the time learning to minimize the chance.

Man pages can be great and people should always read them first. That said, I find that man pages often lack a wide sample of configuration examples, implementation examples and useful tips. This is where blogs and vlogs fill in the gaps. Man pages often leave many things to interpretation that varies wildly from person to person. Man pages sometimes also leave out best practices and caveats.

I think you are dead wrong with this plea. Explanation and reference are two very different things, and both can't exist effectively in the same document. The man pages are great once you have the concepts down. But there's a huge gap out there for introductory explanations of the concepts behind the tools that the man pages (rightfully) assume you already have mastered. This article is a great example of the type of thing we need a lot more of.

If you know you want a SOCKS proxy and just need to know the correct option name, the man pages are great. If you have no idea what a SOCKS proxy is, why you might want it, or even that ssh can solve this problem for you, then there's no chance you'd even think to look at the man pages for ssh and ssh_config, and if you did, you still wouldn't really know what the bits about SOCKS proxy meant for you and your problems.

It's not like this article's existence will prevent people from reading the man pages. On the contrary, this article serves an entirely different purpose. I can read it on my phone, learn about something new, and then when I'm back at my workstation, I can look up the new concept in the man pages to figure out how I can integrate it into my own workflow.

I'm struggling with understanding the motivation behind your post here. Please reconsider the value of this sort of article in combination with the value of man pages.

I find myself agreeing with both of you. You're right that having articles like this makes the tools more accessible and understandable to those that use both resources. However, it seems that most people don't use both resources. Many rely only on these sorts of articles and don't RTFM. There is a lot of power in many *nix tools that people could benefit from but don't realize it because they don't even skim over the manual.

It wasn't that long ago that it was common for people read the manuals of the things they used to get the best bang for their buck, so to speak. Now, the world is filled with quick-starts and use-specific guides, because people don't have the time to research all the things they use.

It's this aversion to a little studying of the tools of our trade that I think blueflow and myself are against.

Cookbooks are also important. Sure, if you learn every single ingredients you might be a good cook... but if you read recipes, you will be able to cook well no matter what.

manpages are such an inconsistent and often wildly disappointing experience that I advise everyone to give up on them and go with <any top internet tutorial> instead.

There are like a half dozen really good manpages I can think of, some others might as well be written in Mandarin

> localhost:~$ ssh remoteserver "cat /var/log/nginx/access.log" | grep badstuff.php

It would be much more traffic (and execution time) friendly, if you put pipe and grep to "" as well, so grep would be executed on remote server and ssh won't have to send the entire log to your machine.

And I'd like to highlight the overwhelming awesomeness of ~/.ssh/config file. If you haven't heard about it or use shell alias instead please please read about it. It supports autocomplete, dynamic paths to key files and many other cool things which can make your ssh client experience simply great.

The article does touch on that optimization:

In this example the grep is being performed on the local system after the log file has been pushed across the ssh session. If the file is large it would be more efficient to run the grep on the remote side simply by enclosing the pipe and grep in the double quotes.

I see no benefit of running grep locally in any case unless maybe local grep is aliased in a complex way to colorize and decorate the outputs in some other way.

Either way this is not a great example as you should just login and run it on the remote machine as running through pipe causes the connection to be established every time unless you keep a persistent connection open behind which is explained in the MultiPlex section which is very handy when you need to run many commands over ssh in cases of like taking backups from that machine.

The obvious case would be when the remote grep doesn't exist at all, or is a cutdown non-gnu version of grep which didn't support things like extended or perl regexps

What chance is there that grep doesn't exist on a machine that has ssh and cat?

Having some undesired grep version could be one reason but not enough to justify piping everything before grep in the example.

There are systems with SSH that are not Unix. For instance, Cisco switches support SSH but don't have a grep. (They do have an equivalent, but it's unfamiliar and likely less feature-rich)

While I myself am not a purist, some people would consider this an instance of UUOC: http://porkmail.org/era/unix/award.html

> localhost:~$ ssh remoteserver "grep badstuff.php /var/log/nginx/access.log"

The only reason I (and probably many other people) prefer cat | grep instead of just using grep is because it is much easier to change the search term this way. With cat, it’s only a ctrl-W away from using a different term. If I only used grep, i would have to move my cursor which can get pretty annoying with a long file path.

That being said, only using grep is my preferred approach (if I don’t forget it) in script and “compact lines” (I.e ssh, watch, and everything that expects the command as a string).

If you are grepping a file multiple times, I would recommend to connect to the server anyways (because it is much faster). That being said, if this is a single occurrence, and saves you from the pitfalls of using the local grep instead of the remote one, your approach is preferable in that case.

> If I only used grep, i would have to move my cursor which can get pretty annoying with a long file path.

You can also do it like this:

    < myfile grep mypattern

$ grep term1

$ ^term1^term2^

Try it

> .ssh/config

Do you mean this or something not described there?

https://nerderati.com/2011/03/17/simplify-your-life-with-an-...

Yes, exactly this.

There is a windows program named sshfs-win[1] that allows mounting of remote SSH location as local folders and SiriKali[2] is currently its only GUI frontend. SiriKali also supports sshfs on Linux.

[1] https://github.com/billziss-gh/sshfs-win

[2] https://mhogomchungu.github.io/sirikali

As an additional tip, I've seen people complain about scp's handling of spaces and other special characters, causing one to use 2 layers of quoting. For example:

  scp machine:'"file with spaces"' .

What people don't tend to realize is that this is because scp allows you to specify the remote-side files with remote shell code. For example, to get all pdfs in the remote home directory:

  scp machine:'*.pdf' .

To get file.xml and file.pdf from the remote:

  scp machine:'file.{xml,pdf}' .

To get the newest file (asumming that it has no whitespace or glob characters):

  scp machine:'$(ls -t | head -1)' .

The same, but handling spaces and other characters, but not newlines:

  scp machine:'"$(ls -t | head -1)"' .

If you use zsh on the remote machine with extended globs, this is the safest way:

  scp machine:'*(oc[1])' .

Is the first "Using a Configuration File" example a bit misleading? The example is:

... Host remoteserver HostName remoteserver.thematrix.io User neo Port 2112 IdentityFile /home/test/.ssh/remoteserver.pub

IdentityFile typically (always?) specifies a file with a private key, right?

Also it's suggested to use ~/.ssh/key to specify a key in your home folder to be a portable config and don't really need to show others what your user name is if it gets shared.

Unless you're using something like a smart card as identity with gpg-agent, then you won't have a private key file. Because of that, you can specify a public key file to identify this private key.

I've been using SSH tunnels for years, this keeps me safe and sane. Also at work where my employer closed almost everything. Including well known web pages with info useful for my work...

I would think twice before violating policy like that. YMMV, of course.

Nice post...covers some good use cases. I wrote a book all about SSH tunneling and port redirection use cases in depth...primarily for penetration testers. "The Cyber Plumber's Handbook" (https://cph.opsdisk.com/) is free (already given away 250+ copies) for students that have an educational/student email address. Simply send an email to cph-student@opsdisk.com and I'll send you a discount code to download it for free.

It walks you through the basics of SSH tunneling (both local and remote port forwards), SOCKS proxies, port redirection, and how to utilize them with other tools like proxychains, nmap, Metasploit, and web browsers.

Advanced topics included SSHing through 4 jump boxes, throwing exploits through SSH tunnels, scanning assets using proxychains and Metasploit's Meterpreter, browsing the Internet through a SOCKS proxy, utilizing proxychains and nmap to scan targets, and leveraging Metasploit's Meterpreter portfwd command.

Does anyone use OSSEC on their personal workstation or personal server? I have some MacOS machines with public IP addresses with ssh access enabled, and I'm wondering if I should use something like this. Presently I have the MacOS firewall configured but that's it.

It's not needed, IMO, if you disable password auth and only allow ssh from white listed IPs. This has eliminated ssh login attempts for me.

I must admit that I have learned 2 new things from this article, i.e.:

- -J to jump through multiple hosts (available since 7.3)

- -R creating a reverse SOCKS proxy (available since 7.6)

New (from a Debian user's perspective) features[1] hide among all the classics.

[1]: https://www.openssh.com/releasenotes.html

I would like to propose the author add a section for chroot sftp and ways to replicate rsync-like behavior, thus allowing rapid file transfer without providing a shell. [1] This is useful in situations where automated file transfers and backups are required and minimizing risk of shell access is desired.

[1] - https://tinyvpn.org/sftp/#lftp

This specific example can be used to set up public semi-anonymous file sharing without providing shell access. there is a working live demo of the sftp server that you can play around with using lftp to replicate rsync like behavior.

Great article! One minor comment though, `folder` sounds so wrong in *nix world, I'd suggest you use directory instead.

Not the author but I find it easier to type folder than directory on qwerty. Not sure about other layout.

I don't think they are really interchangeable terms in this context, one is rigth the other one is wrong

I assume Windows and MacOS use the term folder in place of directory, so at this point, I'm assuming it's generally acceptable to use either one.

macOS, GNOME, KDE, and Xfce use "folders".

Gah, just yesterday I had this crazy problem with SSH tunnels and psql (Postgres client) crashing the tunnel. What solved it? Changing my interface MTU...

http://www.snailbook.com/faq/mtu-mismatch.auto.html

This is interesting. I sometimes have to connect to a VPN that crashes when something provokes a burst of activity (if I cat a file or even ls a large dir). I'll try this, but 576 seems so low; how did you get at this number?

This is what I want to know - the article doesn't explain. I guess its a number that particular (old?) routers / particular firewalls won't fragment the packet..?

Wikipedia says 576 is the minimum for IPv4. I suppose then that the author just went for the value all hosts should support.

I remember as the average packet size on the internet. Googling average packet size yields that- https://etherealmind.com/average-ip-packet-size/

Is there a way to combine screen/tmux with ssh ? like i always need to do some long running tasks over ssh and forget to put it in a screen session, i would like to open each ssh session in a screen session so that when i connect next i should be able to proceed from where i left

This has nothing to do with ssh. That's already shell configuration topic. You can check for a possible solution here:

https://wiki.archlinux.org/index.php/Tmux#Start_tmux_on_ever...

https://superuser.com/questions/224631/is-it-a-good-idea-to-...

Not a gerat solution, but if you did launch something in bash, and you don't care about STDOUT/ERR (or you're directing to a file), you can run "disown %1" to detach it from your shell after you've launched it. The program will continue to run until it finishes naturally (or it's killed)

You can't reattach the program, but there's other ways to auto-launch screen if you're often doing that.

You can indeed reattach a session into screen/tmux:) https://github.com/nelhage/reptyr

Also consider using ssh config to make your default command create or attach a tmux session.

Yes, but this has nothing to do with SSH. You essentially suspend the process, background it, detach it from the parent shell, launch screen/tmux, and re-parent it. Here's a tutorial - http://monkeypatch.me/blog/move-a-running-process-to-a-new-s.... If you ALWAYS want SSH to open in a screen/tmux session, you can probably just alias it on your local system to do so.

Pardon my ignorance. Why are screen/tmux so popular ? I just use a terminal with tabs (i.e. Gnome terminal)

screen/tmux persist when you log out. Let's say you have 10 shells open to a remote server in screen session and you reboot you client computer. If you ssh back into the remote and run screen...pow, you have your 10 shells back.

In part because you can use screen/tmux over an ssh connection.

Few more tips in https://www.commandlinefu.com/commands/matching/ssh/c3No/sor...

Cool post and another remit that half of the software in this world would not have existed if people bothered to learn Unix(s) tools

Very nice article, so many useful things.

Reply on news.ycombinator.com