Hacker News Reader

Comment by rsync | Hacker News Reader

I've been in this business for a long, long time and have come across all manner of innovations regarding network security.

Port knocking (which I think I learned about first at defcon ... perhaps 18 years ago ?) stands out as one of the very few things that made my network(s) substantially safer at nearly zero cost. I love, and continue to love, both the idea and the implementation.

Pay no attention to the nay-sayers and their comical straw-man argument against deploying port knocking by itself. Of course it's not deployed by itself - it is the last layer of defense-in-depth on top of all the other authentication methods you may be doing.

The substantive critique is that you add some amount of complication to your network and introduce a point of failure with knockd - and that is correct. I am happy to report that 15+ years of knockd in production use, all over the world, has produced exactly zero lock-outs or unintentional self-DoS.

View on HN

For me changing the default port for SSH achieved much of the same result at an even lower cost IMO. That is, unless you're high profile enough that attackers might want to target your server specifically and you're not just randomly scanned by botnets which won't generally bother looking beyond port 22.

I had to disable "insecure" ciphers on my bastion SSH hosts at work so that we could pass a security audit. The interesting side effect was that a substantial number of scans failed because the scanning software didn't support them:

  Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512

Put ssh on port 80 and http on port 22 - perfect security ;)

for my internet-facing machines I black-hole everything below 1024 and move SSH to something high but easy enough to remember like 22222

My latest trick for deciding on external ports to forward is to combine the service port and the device's internal IP address. If my desktop was 10.0.1.36 and the service was SSH on port 22, the external port would be 22136. So far it's worked well, but it only really works if you force your router to only give out 10.0.1.2 through 10.0.1.99.

If I was an evil hacker I would look at that port and other variants of 22 like 2200, 2222, etc for people's ssh.

and then all you'd have to do is compromise SSH on my fully patched system

of course the point is not to be able to withstand an attack by a persistent attacker. it prevents my logs from filling up with the constant scans from scripts

Same with RDP. Hackers don't even bother to scan outside of 3389. My logs are empty.

Expanding on that substantive critique, i.e. complexity. Port knocking strips you of the benefits you get from session protocols like TCP.

- Fails when packet loss is high.

- Fails when lags make the attempts arrive in changed order

- Fails when multiple people try to execute the sequence at the same time (yes, you can track source IPs, but then it doesn't work from behind a NAT)

Agreed, none of this is a problem on a small scale. Agreed, you can work around that but it's a PITA.

"Expanding on that substantive critique, i.e. complexity. Port knocking strips you of the benefits you get from session protocols like TCP."

This is correct and worth considering.

In actual practice (again, over 15 years, all over the world, etc.) I have found that the only place I ever run into this is on airplane (in-air) wifi. I don't know if they are specifically blocking random fragments or if they are just too laggy to make this scheme work.

A quick workaround is to ssh to some other host that you control that isn't behind a knock, but does have 'knock' installed and then jump through it.

You could of course just try again if it fails due to these excessively rare cases.

There are a lot of things that fail if packets arrive out of order and with random packet loss your experience is going to be awful, so both of these are rare.

I would argue that port knocking doesn't implement cryptography, but it implements steganography. It's purpose is not to cryptographically secure your service but to conceal its existence.

Can't we do both? Like, the content of the UDP packet is the output of `date -uIs | gpg --sign`.

From knockknock:

> When you want to open a port from a client, you run 'knockknock', which sends a single SYN packet to the server. The packet's IP and TCP headers are encoded to represent an IND-CCA secure encrypted request to open a specified port from the source IP address.

https://moxie.org/software/knockknock/

That page links to a http insecure download script, weird. (Not https, no signature)

This link: wget http://www.thoughtcrime.org/software/knockknock/knockknock-0...

Yeah, the documentation is dated. Not only does it point to an old version, but it would probably be best to get it from the GitHub page he links to. Even then, there are more active forks, so they may be worth checking out to see which would be best to get on board with.

Thanks, seems this is the (also outdated) repo: https://github.com/moxie0/knockknock

Yes I suppose one of the forks is better nowadays

You already do both when you are concealing an ssh server. The cryptography part is the responsibility of the ssh server.

edit: That specific scheme looks naive and susceptible to replay attacks.

The suggestion is nice though since an attacker would have to break both at once. If you conceal an SSH server, you'd first break the obscurity, then proceed with attacking the SSH server. The suggestion seems to be that you would have to break the cryptographic part to reveal the server (which is then supposedly secured by "normal" cryptographic means, such as a regular SSH server). That is, you are dealing with cryptography just to be sure the server exists at all.

I'm all for defense in depth but I don't see the point of a redundant cryptographic layer. It's not trivial that it doesn't broaden the attack surface instead of narrowing it.

If the argument is that a steganographic layer that involves cryptography does a better job at concealing than a similar layer that doesn't I'm also not convinced. Cryptography helps against MITM scenarios, but at that point the existence of the service is already revealed.

What is your conclusion that port knocking has made your network substantially safer based on? Just curious how you measure the effectiveness of something like this.

Could someone point to a contest where an attacker has managed to bypass port knocking. Surely someone has tested its effectiveness at creating more work for attackers.

How can anyone measure the effectiveness of the advice given by a security consultant? It saved a client from disasters that did not happen? How can anyone prove that such disasters would have happened? Answer: Proof of concept.

To question the effectiveness of port knocking, one needs some real stories of real disasters where port knocking was being relied on. Proof of concept. Otherwise all the discussion is still just speculation.

from personal experience, simply moving ssh off of port 22 cuts your drive-by log noise by 99.9%, which makes auditing clean logs much much easier.

failed login attempts with ssh on 22: hundreds per week, after: 0. i imagine with knockd it would go negative.

I wonder if it would be interesting to load a ssh honeypot that would feed from the logs so it would add the logins and then just dump the ip + input to a file.

Honeypot could basically run in a some sort of isolation layer (like Sandboxie or jails) and then self-destruct after the automated script is gone... and then you slam the door on that user/ip combo for good.

I can't help but think this would be interesting...

One of my university courses offered an opportunity for a project like this and I did it with some classmates.

We started by altering the ssh daemon to disallow all logins over this ssh daemon and to log all the usernames and passwords attempted. After a week we gathered thousands of attempts to brute force into the honeypot. Interestingly enough, the passwords used were a combination of the very commonly used ones but also ones that were clearly from other popped boxes.

After a week or so of this we altered the ssh daemon again. This time it would log all attempts but also grant access on the 3rd attempt no matter what the credentials were. The few bots that managed to get in all tried to install various rootkits on the machine, all of which where targeted at a different distro of Linux than we were using so it mostly just busted up our shells output.

It was a long time ago but a great experience.

Fail2ban [0] bans ip addresses based on failed login attempts (works for more than ssh), minus the isolation layer.

[0]: https://www.fail2ban.org/wiki/index.php/Main_Page

fail2ban should be avoided. It does not support IPv6, so should be considered legacy software.

EDIT: Source: https://github.com/fail2ban/fail2ban/issues/1123

It appears they have moved forward a little in supporting IPv6, but it's still incomplete. It's unacceptable to not support it fully in 2020.

It looks like ipv6 matching is supported since late 2017 (version 10.0 [0]), although the changelog states that "not all ban actions are IPv6-capable now". As for IPv6 capabilities, I don't have any recent experience with the software.

[0]: https://github.com/fail2ban/fail2ban/blob/0.11.1/ChangeLog

With IPv6 every user gets an IPv4 internet worth of addresses for himself which makes fail2ban useless.

Couldn't you just ban the /64 and call it good? It's not like they get a random selection of addresses, they're all going to be the same CIDR. Or am I overlooking something here?

it does more than just scanning iptables logs..

> then just dump the ip + input to a file.

There are people who do just that.. and then try connect back to that IP with the same credentials. Easy way to find some shells..

I think you're talking about something like Kippo: https://www.honeynet.org/projects/old/kippo/

You don't even need a honeypot, simply ban ip addresses for some time after X failed login attempts.

The point of the honeypot is that there's no heuristic causing any delay or elusive attacker being missed (e.g botnet trying once per IP). You don't even need any processing time, nor even complete syn/synack/ack: any TCP connection attempt to that port triggers an instaban.

> i imagine with knockd it would go negative.

So it's the people running knockd that are actually the ones generating all these ssh login attempts...

Wouldn't a reasonable rebuttal be that the log filtering rule that discards these failed login attempts would accomplish the same security goal, with less mechanism?

i prefer knowing that nothing is happening rather than filtering things i dont like.

also, lots of people sing the praises of fail2ban, which needs to watch log files to operate, but you can get most of its utility by simply rate-limiting using iptables' session tracking without any additional mechanisms.

https://selivan.github.io/2018/07/07/limit-new-connections-f...

I can tell you right now what's happening and compress all those log records into a single record:

Every exposed 22/tcp on the Internet is being continuously exposed to automated SSH scanners running from thousands of points on the Internet. There is nothing you can reasonably do to prevent it (the sources are so diverse you couldn't even realistically block a determined scanner), and, if you've turned off password authentication --- which you must do anyways --- the probes aren't a meaningful threat.

From a systems-administration standpoint, it's nice to be able to easily sift out the random bots and drive-bys from the more personal probes that maybe have a human at the other end.

There are myriad bits and bobs of software and services that promise to do anomaly detection in logs, but an easier approach for now is to just move ssh off 22. It affords basically zero extra real security but makes any subsequent login attempts more worthwhile to look at.

Let me gently challenge you: you've moved SSH to 7163/tcp. You see a pattern of login attempts. Now what, and why?

Sure. Direct answer: search logs across all services and nodes on the network for activity from that source or netrange. See if it's targeting particular usernames, especially anything that shouldn't be guessable. Treat it first as a source of information: did I screw something up somewhere? Have we had an incident I don't know about? If it looks like nothingburger, ignore it and get on with the day.

Broader answer after reading some of your other comments in this thread: in your ideal, highly secured network environment, none of this is necessary because everything's wired up tighter than a gnat's ass hole. Unfortunately I've never had the pleasure of working for one of those places.

I guess that's fair. But: if you buy into this idea, there's a much, much better thing to do: look into Canarytokens. Canarytokens, unlike port knocking, are criminally underused. Really, you should do something similar for your off-port SSH service; don't actually _run_ SSH there, just run a stateless unprivileged service that spoofs a bit of SSH protocol and generates loud alerts.

The networks I'm describing aren't "ideal" or "highly secured". I am describing table stakes. While I was at Latacora, most of the clients we engaged with were already at this level of maturity when we joined up.

Cool, thanks. I found https://blog.thinkst.com/p/canarytokensorg-quick-free-detect...; is there a better writeup you have offhand? https://github.com/cowrie/cowrie has been on my todo list for a while too, but more as a toy than a serious thing.

> The networks I'm describing aren't "ideal" or "highly secured". I am describing table stakes. While I was at Latacora, most of the clients we engaged with were already at this level of maturity when we joined up.

One of the most valuable things you do here is describe things that you believe are table stakes to people and organizations that have never heard of them. Companies like Latacora tend to self-select for clients that are at least aware that security should be a sensible line item in their quarterly budget. There are many many more organizations for whom moving ssh or even port knocking amount to a real improvement to their infrastructure. :-(

There's:

https://canary.tools/help/canarytokens

But the concept is really simple: come with any kind of thing you'd want to tripwire --- the AWS key is particularly slick --- and put it somewhere in your infra, then wait for alerts.

What happens when the attacker modifies /etc/resolv.conf, disables or does not use DNS while on the network.

What if the attacker does not use software that automatically loads images, like the "canary token" tracking pixels?

What is the attacker opens the files offline?

Canary tokens still work? How?

Why would anyone advise using canary token instead of port knocking? Makes no sense. Why not use both?

To defend Canarytokens here (and I’m totally biased because we make it) some tokens can’t be easily avoided. Ie. If the token is a Slack/AWS/something API key, then the only way for the attacker to profit is to use it, and the moment they do, they tip their hand. The joy of Canarytokens is not having to set up infrastructure to get the alerting win, with very little effort.

Not "knocking" the idea but to be honest using someone else's API key and expecting that no one is going to notice sounds really dumb. I guess if that is the level of intelligence you are up against, then "winning" can indeed be quite easy. Although I would argue if the goal is to restrict access and they managed to gain access then regardless of what they do next, whether it is smart or stupid, they have "won".

I don't even understand the question. We were talking about the value of moving SSH to a different port to cut down on logged probes. That at least has the value of giving you a weak signal about IP sources that are somewhat determined to break in. Port knocking doesn't even do that.

Maybe it is not supposed to. Port knocking is not a substitute for anything. Yet most criticisms of it, like yours, seem to assume it is going to be used as a replacement for something else.

> the sources are so diverse you couldn't even realistically block a determined scanner

thankfully, i dont have properties which attract determined scanners. and i imagine that's true for 99% of public ip addresses, too.

So then, of course, there isn't even a theoretical reason why you'd want to pay attention to SSH probes.

i mean, you're in the security industry, where targeted attacks are the default assumption. the overwhelming majority of web properties are not valuable enough to get this kind of attention.

even if security through obscurity is not real protection from targeted attacks, it is at minimum significant noise reduction, and quite possibly a reasonable barrier against drive-bys which assume default configurations. i'm sure you know how many scanners try to access some default php/wordpress admin route. how much of that would disappear if wordpress was simply installed behind a random 32-char route instead of /wp-admin? all it is is obscurity, sure. but is it not security?

The whole "obscurity isn't security" thing is a super interesting topic. Suppose you have a static file on a public web server and you want to control access to the file... confidentiality. Many people would agree that if you come up with a long string, don't disclose it to untrusted parties, and use it as a basic auth password, then you have secured the file to some extent. And many people would claim that if you take that same undisclosed string, and make it a path segment (parent directory) for the file, then you have merely obscured the file, or perhaps secured it to a far lesser extent. The former makes it "not available to the public" while the latter makes it "available to the public, they just don't know where to look."

I call BS. These are pretty much the same thing* which is to say: an adversary trying to get the file needs to know something that they don't know, and can't reasonably guess/discover any time soon. Did you secure it or obscure it? Whatever you call it, you require a "something you know" factor.

*Obviously, there's the fact that other parts of the stack -- browser history, access logs, rate limiting, etc. -- actually treat a password payload with the respect it deserves and assume there's nothing sensitive about a URI path... kindly pretend my example doesn't have this flaw.

Semantics of this is wildly different.

You may accidentally enable the directory listing.

While you may say that certain methods of withholding the information are isomorphic at certain conditions it doesn’t mean that they are semantically equivalent.

I admit that my example is flawed for a few reasons and autoindex is another in that list. While obscurity and security are not identical by any means, and therefore not semantically equivalent in general, I'm just pointing out how the "something you know" factor inherently utilizes what common ground they do share, whether you call it a secret, a key, a password, a URL with a long random portion.

Topically relevant (and amusing) username!

http://bash.org/?244321

Turn on logging for failed connection to port 22 and you have just recreated the same log file, assuming you trust the underlying security in your sshd configuration. You just moved the "nothing happened" event from the sshd to the firewall, and the firewall by default do not log such events.

The best argument in favor of knockd that I can think of is the same of fail2ban, which is that ssh connection is actually a finite resource. You can only have so many simultaneous TCP connections. Rejecting packages in the firewall has no limits, and I have had servers which has received service disruption in the past because swarms of ssh bots used up the available tcp connections that the web server could handle. fail2ban in my case was a less intrusive change to our work flow so we use that rather than knockd to solve the problem.

I will add as a small note that the default of not logging firewall events has come to bite us once in a while. Just like keeping the default in ssh to log, keeping the default to not log is a trade off between resource usage and keeping logs of events.

> simply moving ssh off of port 22 cuts your drive-by log noise by 99.9%, which makes auditing clean logs much much easier.

Is there a reason you can't use some form of IP whitelisting?

I'm not a proper sysadmin, but when I play about on EC2 I always configure the security-group to block incoming connections to TCP port 22 except from whitelisted IPs.

The only time I've had to broaden the whitelist beyond just a few static IPs, is with mobile Internet tethering. They seem to frequently change my IP, so I guessed at my provider's IP range and whitelisted it with a pretty broad wildcard mask. Still much better than accepting connections from any old address.

I can see that my rather simple approach might not scale very nicely.

> I'm not a proper sysadmin, but when I play about on EC2 I always configure the security-group to block incoming connections to TCP port 22 except from whitelisted IPs.

Because I have multiple IPs I connect to, most of which are unknown. Maybe I'm connecting from a coffee shop, maybe from a tethered phone, maybe from work, maybe from an airplane, or a train, or a friend's wifi, etc.

There is this idiotic notion that any security measure that cannot stand on its own is "security by obscurity", and therefore useless.

That principle is taught in school and echoed by every junior netsec professional out there.

After almost 20 years of work, I can say that these layered techniques, are invaluable. OF COURSE, the core security has to be bulletproof. But once you've covered that base, there is an enormous value in keeping your assets under the radar.

At the very least, it reduces the number of alerts you need to review by several orders of magnitude. Logs become READABLE.

any tips or where one would get best practice for configuring/setting up/using knockd?

I wrote up how having a server on the internet is scary now (http://redgreenrepeat.com/2020/03/20/why-you-should-secure-y...) and how to protect it (http://redgreenrepeat.com/2020/04/10/how-to-secure-your-serv...)

One thing I didn't get into more was port knocking/knockd as there were not many resources for it.

I'd love to learn more about how you have used knockd.

My default ipfw ruleset is very restrictive - no ports open.

The command that gets run when the (correct) knock comes in is an ipfw command:

  /sbin/ipfw add 01021 allow tcp from %IP% to 10.0.0.10 22,443 setup

... so now the knocking IP can see TCP 22 and 443 (and nothing else).

I then have a cron job that runs every night at midnight that deletes those rules:

  0 0 * * * /sbin/ipfw delete 01021

... so I am required to knock daily.

The following is a little lame, but I want to see who has knocked so far today (should just be me and my own IPs) so I do this every minute:

  * * * * * /sbin/ipfw show|/usr/bin/grep ^01021 | awk '{print $7}' > /tmp/.knock_list

... and then cat that list to myself every time I log in ...

So with poet knocking, if I only have an ssh port open and a couple of ports my servers server content on like an http and https server, I'm assuming id use port knocking to get inside my ssh port right? I don't need to do port knocking for my http and https ports because they're open but not as an ssh service?

Yes, that's right - you'd leave your public services open by default and the knock would simply open up tcp22 ...

> So with poet knocking

Knocking a poet is bound to get you into trouble. Poets fight back, and hard. Just don't do it.

But they're just so damn eloquent all the time. It drives me nuts.

Moxie actually published a nice lighweight port knocking daemon that:

* Does not bind to sockets

* Is not written in C

* Prevents replay attacks

* The only code that runs as root is just tailing kern.log and is like 15 lines

It probably needs to roll from SHA1 to SHA256 and Python2 to Python3, but otherwise seems sound.

[1] https://moxie.org/software/knockknock/

Looks like there is a pull request to change it from authenticate-then-encrypt to encrypt-then-authenticate that hasn't been merged for over five years. Thinking this project might be abandoned, unfortunately. :-(

Edit: It appears there may be an active fork here: https://github.com/indyprime/knockknock

Still digging through to see if the encrypt-then-authenticate change made it into this version.

Edit 2: It looks like it still uses authenticate-then-encrypt. I sent an email to the maintainer of that fork, we'll see what happens. ^__^

This is freaking genius!

Personally I avoid knockd and instead use something like fwknop https://github.com/mrash/fwknop. You get the advantage of port kocking while protecting yourself from re-play attacks. The main dis-advantage is it is a more complicated procedure so it may limit you a bit.

How I set it up specifically: https://peekread.info/tech/20190513-fwknop/

vpn + port knocking should be good enough but I am not sure how to implement it in an efficient way (I currently only use a VPN)

https://www.howtogeek.com/105693/how-to-knock-into-your-netw...

Does this have any advantages over using plain wireguard? WG operates purely over UDP and doesn't respond unless you send it packets with an authorized key, so it's essentially the VPN and port knocking all in one.

I guess if you only have the key, you would still need to figure out how to use it

I think "attacker has a private key" is an unreasonable threat model to protect against, not least because the key is so much harder to crack than port knocking. The benefit to port knocking is against information disclosure ("this server is running Apache httpd version x.y and sshd version z"), brute force, noisy logs, and pre-auth vulnerabilities (things like heartbleed and shellshock). While in theory I suppose wireguard could be affected by pre-auth vulnerabilities and maybe a completely blind bruteforce attack, it's a listening UDP port that doesn't respond until it sees valid credentials, so it's completely invisible to an attacker.

There's a similar , very sneaky tool: sslh. Practically no automated tools will attempt to SSH into the HTTPS port...

I presume this connects SSH clients, and serves a fake web page to browsers?

Sounds pretty cool, but do SSH clients (e.g. PuTTY need special support for it)

Nope, no support needed. SSLH will proxy SSL traffic to whatever other SSL service you have running (nginx, openvpn, etc.), and send the SSH traffic to openssh. It look like it even supports recognizing plain HTTP traffic as well and proxying it accordingly.

I've been wanting to implement knockd on something for years but people always push back. I don't actually think it's incredibly useful but I love the cool factor. Good for you for actually doing it.

What do you think about single packet authorization?

Reply on news.ycombinator.com