Port knocking
en.wikipedia.org
en.wikipedia.org
Port knocking (which I think I learned about first at defcon ... perhaps 18 years ago ?) stands out as one of the very few things that made my network(s) substantially safer at nearly zero cost. I love, and continue to love, both the idea and the implementation.
Pay no attention to the nay-sayers and their comical straw-man argument against deploying port knocking by itself. Of course it's not deployed by itself - it is the last layer of defense-in-depth on top of all the other authentication methods you may be doing.
The substantive critique is that you add some amount of complication to your network and introduce a point of failure with knockd - and that is correct. I am happy to report that 15+ years of knockd in production use, all over the world, has produced exactly zero lock-outs or unintentional self-DoS.
I wrote up how having a server on the internet is scary now (http://redgreenrepeat.com/2020/03/20/why-you-should-secure-y...) and how to protect it (http://redgreenrepeat.com/2020/04/10/how-to-secure-your-serv...)
One thing I didn't get into more was port knocking/knockd as there were not many resources for it.
I'd love to learn more about how you have used knockd.
https://www.howtogeek.com/105693/how-to-knock-into-your-netw...
How I set it up specifically: https://peekread.info/tech/20190513-fwknop/
The command that gets run when the (correct) knock comes in is an ipfw command:
/sbin/ipfw add 01021 allow tcp from %IP% to 10.0.0.10 22,443 setup
... so now the knocking IP can see TCP 22 and 443 (and nothing else).I then have a cron job that runs every night at midnight that deletes those rules:
0 0 * * * /sbin/ipfw delete 01021
... so I am required to knock daily.The following is a little lame, but I want to see who has knocked so far today (should just be me and my own IPs) so I do this every minute:
* * * * * /sbin/ipfw show|/usr/bin/grep ^01021 | awk '{print $7}' > /tmp/.knock_list
... and then cat that list to myself every time I log in ...Knocking a poet is bound to get you into trouble. Poets fight back, and hard. Just don't do it.
* Does not bind to sockets
* Is not written in C
* Prevents replay attacks
* The only code that runs as root is just tailing kern.log and is like 15 lines
It probably needs to roll from SHA1 to SHA256 and Python2 to Python3, but otherwise seems sound.
Edit: It appears there may be an active fork here: https://github.com/indyprime/knockknock
Still digging through to see if the encrypt-then-authenticate change made it into this version.
Edit 2: It looks like it still uses authenticate-then-encrypt. I sent an email to the maintainer of that fork, we'll see what happens. ^__^
failed login attempts with ssh on 22: hundreds per week, after: 0. i imagine with knockd it would go negative.
also, lots of people sing the praises of fail2ban, which needs to watch log files to operate, but you can get most of its utility by simply rate-limiting using iptables' session tracking without any additional mechanisms.
https://selivan.github.io/2018/07/07/limit-new-connections-f...
Every exposed 22/tcp on the Internet is being continuously exposed to automated SSH scanners running from thousands of points on the Internet. There is nothing you can reasonably do to prevent it (the sources are so diverse you couldn't even realistically block a determined scanner), and, if you've turned off password authentication --- which you must do anyways --- the probes aren't a meaningful threat.
thankfully, i dont have properties which attract determined scanners. and i imagine that's true for 99% of public ip addresses, too.
even if security through obscurity is not real protection from targeted attacks, it is at minimum significant noise reduction, and quite possibly a reasonable barrier against drive-bys which assume default configurations. i'm sure you know how many scanners try to access some default php/wordpress admin route. how much of that would disappear if wordpress was simply installed behind a random 32-char route instead of /wp-admin? all it is is obscurity, sure. but is it not security?
I call BS. These are pretty much the same thing* which is to say: an adversary trying to get the file needs to know something that they don't know, and can't reasonably guess/discover any time soon. Did you secure it or obscure it? Whatever you call it, you require a "something you know" factor.
*Obviously, there's the fact that other parts of the stack -- browser history, access logs, rate limiting, etc. -- actually treat a password payload with the respect it deserves and assume there's nothing sensitive about a URI path... kindly pretend my example doesn't have this flaw.
You may accidentally enable the directory listing.
While you may say that certain methods of withholding the information are isomorphic at certain conditions it doesn’t mean that they are semantically equivalent.
There are myriad bits and bobs of software and services that promise to do anomaly detection in logs, but an easier approach for now is to just move ssh off 22. It affords basically zero extra real security but makes any subsequent login attempts more worthwhile to look at.
Broader answer after reading some of your other comments in this thread: in your ideal, highly secured network environment, none of this is necessary because everything's wired up tighter than a gnat's ass hole. Unfortunately I've never had the pleasure of working for one of those places.
The networks I'm describing aren't "ideal" or "highly secured". I am describing table stakes. While I was at Latacora, most of the clients we engaged with were already at this level of maturity when we joined up.
> The networks I'm describing aren't "ideal" or "highly secured". I am describing table stakes. While I was at Latacora, most of the clients we engaged with were already at this level of maturity when we joined up.
One of the most valuable things you do here is describe things that you believe are table stakes to people and organizations that have never heard of them. Companies like Latacora tend to self-select for clients that are at least aware that security should be a sensible line item in their quarterly budget. There are many many more organizations for whom moving ssh or even port knocking amount to a real improvement to their infrastructure. :-(
https://canary.tools/help/canarytokens
But the concept is really simple: come with any kind of thing you'd want to tripwire --- the AWS key is particularly slick --- and put it somewhere in your infra, then wait for alerts.
What if the attacker does not use software that automatically loads images, like the "canary token" tracking pixels?
What is the attacker opens the files offline?
Canary tokens still work? How?
Why would anyone advise using canary token instead of port knocking? Makes no sense. Why not use both?
The best argument in favor of knockd that I can think of is the same of fail2ban, which is that ssh connection is actually a finite resource. You can only have so many simultaneous TCP connections. Rejecting packages in the firewall has no limits, and I have had servers which has received service disruption in the past because swarms of ssh bots used up the available tcp connections that the web server could handle. fail2ban in my case was a less intrusive change to our work flow so we use that rather than knockd to solve the problem.
I will add as a small note that the default of not logging firewall events has come to bite us once in a while. Just like keeping the default in ssh to log, keeping the default to not log is a trade off between resource usage and keeping logs of events.
Honeypot could basically run in a some sort of isolation layer (like Sandboxie or jails) and then self-destruct after the automated script is gone... and then you slam the door on that user/ip combo for good.
I can't help but think this would be interesting...
We started by altering the ssh daemon to disallow all logins over this ssh daemon and to log all the usernames and passwords attempted. After a week we gathered thousands of attempts to brute force into the honeypot. Interestingly enough, the passwords used were a combination of the very commonly used ones but also ones that were clearly from other popped boxes.
After a week or so of this we altered the ssh daemon again. This time it would log all attempts but also grant access on the 3rd attempt no matter what the credentials were. The few bots that managed to get in all tried to install various rootkits on the machine, all of which where targeted at a different distro of Linux than we were using so it mostly just busted up our shells output.
It was a long time ago but a great experience.
EDIT: Source: https://github.com/fail2ban/fail2ban/issues/1123
It appears they have moved forward a little in supporting IPv6, but it's still incomplete. It's unacceptable to not support it fully in 2020.
[0]: https://github.com/fail2ban/fail2ban/blob/0.11.1/ChangeLog
There are people who do just that.. and then try connect back to that IP with the same credentials. Easy way to find some shells..
So it's the people running knockd that are actually the ones generating all these ssh login attempts...
Is there a reason you can't use some form of IP whitelisting?
I'm not a proper sysadmin, but when I play about on EC2 I always configure the security-group to block incoming connections to TCP port 22 except from whitelisted IPs.
The only time I've had to broaden the whitelist beyond just a few static IPs, is with mobile Internet tethering. They seem to frequently change my IP, so I guessed at my provider's IP range and whitelisted it with a pretty broad wildcard mask. Still much better than accepting connections from any old address.
I can see that my rather simple approach might not scale very nicely.
Because I have multiple IPs I connect to, most of which are unknown. Maybe I'm connecting from a coffee shop, maybe from a tethered phone, maybe from work, maybe from an airplane, or a train, or a friend's wifi, etc.
How can anyone measure the effectiveness of the advice given by a security consultant? It saved a client from disasters that did not happen? How can anyone prove that such disasters would have happened? Answer: Proof of concept.
To question the effectiveness of port knocking, one needs some real stories of real disasters where port knocking was being relied on. Proof of concept. Otherwise all the discussion is still just speculation.
Sounds pretty cool, but do SSH clients (e.g. PuTTY need special support for it)
> When you want to open a port from a client, you run 'knockknock', which sends a single SYN packet to the server. The packet's IP and TCP headers are encoded to represent an IND-CCA secure encrypted request to open a specified port from the source IP address.
This link: wget http://www.thoughtcrime.org/software/knockknock/knockknock-0...
Yes I suppose one of the forks is better nowadays
edit: That specific scheme looks naive and susceptible to replay attacks.
If the argument is that a steganographic layer that involves cryptography does a better job at concealing than a similar layer that doesn't I'm also not convinced. Cryptography helps against MITM scenarios, but at that point the existence of the service is already revealed.
- Fails when packet loss is high.
- Fails when lags make the attempts arrive in changed order
- Fails when multiple people try to execute the sequence at the same time (yes, you can track source IPs, but then it doesn't work from behind a NAT)
Agreed, none of this is a problem on a small scale. Agreed, you can work around that but it's a PITA.
There are a lot of things that fail if packets arrive out of order and with random packet loss your experience is going to be awful, so both of these are rare.
This is correct and worth considering.
In actual practice (again, over 15 years, all over the world, etc.) I have found that the only place I ever run into this is on airplane (in-air) wifi. I don't know if they are specifically blocking random fragments or if they are just too laggy to make this scheme work.
A quick workaround is to ssh to some other host that you control that isn't behind a knock, but does have 'knock' installed and then jump through it.
of course the point is not to be able to withstand an attack by a persistent attacker. it prevents my logs from filling up with the constant scans from scripts
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512That principle is taught in school and echoed by every junior netsec professional out there.
After almost 20 years of work, I can say that these layered techniques, are invaluable. OF COURSE, the core security has to be bulletproof. But once you've covered that base, there is an enormous value in keeping your assets under the radar.
At the very least, it reduces the number of alerts you need to review by several orders of magnitude. Logs become READABLE.
I lost my fob. So I took my intercom apart and wired a band pass filter circuit between the buzzer input and actuator output. If you pressed my buzzer with the right steady pattern, it would automatically let you in.
It worked great and I didn’t buy a replacement fob.
A few months later I started getting woken up by someone buzzing my buzzer. They would always scurry in before I could open my door to see who it was.
Turns out my upstairs neighbor saw my shenanigans a few times and started freeloading off my invention. Honestly amazed he/she reverse engineered what was going on!
Needless to say, I bought a new fob.
I understand the key fob/card concept for access purposes, but have never heard of an “intercom” in the context of home/apartment/etc security.
First friend-a local-is the type who doesn’t answer his phone from numbers he doesn’t know/isn’t saved as a contact-which I knew, so I dialed him, then dialed his wife (who also doesn’t answer unknown numbers), then dialed him again. He picked up the second time and after some explaining he chuckled and goes “yeah I figured it must be important when the same number called $wife immediately after calling me”.
He was there to pick us (including drunk mutual friend) up within the hour. And yes, he’s a network engineer heh.
Not sure how payment worked and why you could listen for free—I'm only familiar with the method from written accounts.
The other person would get an option to receive a call from <whoever> for $5, but usually <whoever> is the name or code of a location to be picked up.
I must have done this 100 times as a kid. Once an angry man clued onto it, switched my line across and told me off. This is back in the days where mobile calls were expensive and very few kids had phones.
But you just gave me an idea for something I'll probably never implement: an audio key. The intercom auto-responds and if the audio matches the key (hello 90s modem sounds), then it lets you in. You could add more automation on top and provide temporary keys for guests, etc. Just play the audio you received on whatsapp, it will open the door. Please someone implement this and get rich doing it so that I can just buy it. It needs to take into account when you're at home and you still want to answer the mailman.
In 2020, it's gone past "silly" and is now a design smell, in the same way that actually relying on fail2ban would be. Whatever port knocking is doing to help you is a a description of something you're doing wrong with your architecture.
We're not even comfortable with long-lived keys anymore; modern networks are built with short-lived certificates issued off 2FA from an IdP. We have good VPNs right now for the other weird cases. Our networks default to all-ports all-hosts closed. Arguably, port knocking is literally setting your security back, by making it harder for automated tooling to scream bloody murder if you've managed to expose a dev instance unexpectedly to the Internet.
Don't bother. If you've got a single system you're managing outside of a cloud provider with its own decent network firewall, just set up WireGuard and filter everything but that. If it helps, think of WireGuard as the evolution of SPA port knocking.
What are the chances of the cracker finding the exact combination of ports to knock on, and then the exact key to use? If they're able to do that then they are a dedicated attacker and they're mitm'ing a presumably encrypted connection on your network, so you probably have more problems than just one bad machine, they already replicated or replaced a certificate on your machine, or worse.
And if you then expose a service that isn't via the port knocking algorithm, surely that would show up?
> We're not even comfortable with long-lived keys anymore
Can you not just change the key N number of logins and factor in 2FA as well?
I'm not sure how port knocking cannot, at least theoretically, be used in conjunction with the other measures you talked about.
Don't do that.
In AWS (for example), have a dev VPC, and a WireGuard forwarder on it (or, if you're doing things 2010-style, an SSH bastion). If the machine needs to expose 443 directly to the Internet, give it an EIP; have the VPC prevent traffic to anything but 443 (and WireGuard).
What, in that architecture, would port knocking even be accomplishing? Keeping the wrong developer off your machine?
As for how attackers would break port knocking: if you believe attackers can't passively watch traffic, either directly or through redirection, I have good news for you about how much money you need to spend on encrypting traffic!
You do not give a single argument on why the complex thing is meaningfully safer in practice.
If security is equal for all practical purposes in either case, this would be the wrong thing to do, because it is easier to fuck up complex things and it's more expensive to manage them.
Oh shit, I just gave away my secret on a public forum...
Ha ha, fooled you! I use a different secret letter in actual practice!
(Or do I?)
p.s. Pretty sure port-knocking offers less protection than this scheme.
When the next 0day hits… will it be enough to protect them, yes. After all, they’re not a target and the automated attacks won’t affect them.
In my case, all internal services in a kubernetes cluster have no path that allow it to be accessed from the outside. Occasionally, developers might want to do so, say to access a database or prometheus server.
The way to properly do this, imo, is to use a hardened service that functions as a networking proxy. Use proper encryption keys for access, and it should be quite hard to bust open.
Where does port knocking come in? That service should need to be exposed at all times. So, use port knocking to configure the infrastructure to temporarily allow access from your IP, to that proxy servers IP, on that particular port.
I'm more inclined to not take it serious, if such a simple mechanism isn't in place to protect infrastructure.
Modern networks are sadly far too rare; 95%+ of the industry is built on legacy networks.
My take on fail2ban and port knocking are:
* they reduce the amounts of logs produced
* they defend against fully automated at-scale scans / password stuffing attacks that a well-maintained application wouldn't need defending against, but we all know that in any sufficiently big organization, not all applications and accounts will be well maintained.
It is possible to do similar things with haproxy. Configured to listen for TLS connections on a large number of ports, look for a secret combination of custom headers and values, then, if found and matching, forward to a localhost ucspi tcpserver on the backend. The tcpserver may then execute some program, for example sshd or pfctl.
Here's an example of what that would look like:
https://gist.github.com/dcorbett-haproxy/ec7059cbfccf12c8f41...
Take a look at fwknop for the implementation. The only issue with it is there’s no easy install for pfsense.
https://wiki.archlinux.org/index.php/Port_knocking#Port_knoc...
Here’s my example for a VPN running on OpenWrt. If you experience any race conditions with iptables you can pepper each rule with something like “-w 5”
This opens Wireguard port 666 for 15 seconds. I have a script that creates my ipset allowing connections from the USA only.
# The correct port sequence is 1111 -> 2222 -> 3333 -> 4444; any other sequence will drop the traffic
iptables -N WG-INONE
iptables -N WG-INTWO
iptables -N WG-INTHREE
#
iptables -A input_wan_rule -m conntrack --ctstate NEW -m udp -p udp --dport 666 -m set --match-set usa src -m recent --mask 255.255.255.0 --rcheck --name WG3 --seconds 15 -j ACCEPT
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp -m recent --mask 255.255.255.0 --name WG3 --remove -j DROP
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp --dport 4444 -m recent --mask 255.255.255.0 --rcheck --name WG2 -j WG-INTHREE
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp -m recent --mask 255.255.255.0 --name WG2 --remove -j DROP
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp --dport 3333 -m recent --mask 255.255.255.0 --rcheck --name WG1 -j WG-INTWO
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp -m recent --mask 255.255.255.0 --name WG1 --remove -j DROP
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp --dport 2222 -m recent --mask 255.255.255.0 --rcheck --name WG0 -j WG-INONE
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp -m recent --mask 255.255.255.0 --name WG0 --remove -j DROP
iptables -A input_wan_rule -m conntrack --ctstate NEW -m tcp -p tcp --dport 1111 -m recent --mask 255.255.255.0 --name WG0 --set -j DROP
iptables -A WG-INONE -m recent --mask 255.255.255.0 --name WG1 --set -j DROP
iptables -A WG-INTWO -m recent --mask 255.255.255.0 --name WG2 --set -j DROP
iptables -A WG-INTHREE -m recent --mask 255.255.255.0 --name WG3 --set -j DROP
EDIT - For those wondering about the Netmask, it's for mobile connections.Problems included various operating systems and network tools knocking in ways that iptables hates but a knock daemon will handle (e.g. sending two requests per port); making iptables rules unnecessarily complex and hard to audit and eventually conflicting with other things that interacted with iptables; and, once, accidentally getting the knock-related rules nuked by some other software and turning a simple login into a fun little puzzle at exactly the wrong time to be solving more puzzles.
We tested both approaches early on. A working knockd implementation cost less than an hour, including getting other users set up to interact with it. The iptables implementation he switched to shortly after ended up costing dozens of hours.
If I were to use a daemon I’d go with an SPA, like fwknop. The lack of an iOS client is the only reason I still use port knocking.
It's not stupid to have a tiny tiny service to unlock your bigger attack surfaces. It IS stupid to implement "IP-over-SYNpackets" (actually "password over SYN packets") when we already have a perfectly good way to send packets of information that doesn't expose even the complexities of TCP (e.g. slowlaris, SYN flood, etc…). And it's called UDP. Just send your plaintext password in a UDP packet.
It's not exactly hard to safely receive a UDP packet. In fact it's orders of magnitude easier to make a UDP server secure than a raw packet sniffer.
But also I would argue that TCP MD5 (RFC 2385) is vastly underused. It doesn't work through NAT, so maybe now that IPv6 is becoming more and more common it'll gain more traction. It's been used in production with BGP since forever (at least since the 90s), but there's nothing stopping it being used for SSH too. (in fact I run a patched OpenSSH that uses TCP MD5, see my blog post)
TCP MD5 also prevents your connection from being reset on a TCP level, once it's up and running, even by someone who can inject spoofed packets and sniff your encrypted connection.
More on this: https://blog.habets.se/2019/11/TCP-MD5.html
``` for x in 9000 8000 7000; do nmap -Pn --host_timeout 201 --max-retries 0 -p $x server_ip_address; done ```
It seems like you would get the same benefits by moving all major at-risk services (eg SSH) to non-standard ports, and set up exponential IP bans on attempts to communicate on any other port.
I have used it in the past and has saved me a few times. This is only on the bastion.
For actual security, though, it’s also woefully inadequate:
1. It adds very little security. How many bits are in a “secret knock”?
2. The security it does add is bad: It’s sent in cleartext, and easily brute-forced.
3. It complicates access, since it’s non-standard.
Assuming you use most of the 2^16 ports, and a typical text password is based on an alphabet of ~70 characters, each knocking step is roughly equivalent to log_70(2^16) ≃ 2.61 traditional password characters.
For example, 5 steps of knocking already entail (2^16)^5 ≃ 1.2*10^24 steps to check. With 1000 attempts per second, that still means trying > 10^13 years until all combinations have been tried.
It’s still trivially sniffed, since it’s sent in the clear.
And it’s hard to change, since you have to change all the places which needs it.
https://github.com/jftuga/gofwd
gofwd is a cross-platform TCP port forwarder with Duo 2FA and Geographic IP integration. Its use case is to help protect services when using a VPN is not possible. Before a connection is forwarded, the remote IP address is geographically checked against city, region (state), and/or country. Distance (in miles) can also be used. If this condition is satisfied, a Duo 2FA request can then be sent to a mobile device. The connection is only forwarded after Duo has verified the user.
Thoughts?
We are now using the same technique in a network security product at AppGate which is used to protect highly sensitive workloads around the world. https://www.appgate.com/blog/dont-be-fooled-by-lowercase-sdp...
Port Knocking is great, another layer of defense.
Had it on a openbsd box, firewall script deamon would monitor the firewall logs, pick up the sequence and open ssh for that ip if matched and as sequence was S/KEY, it was always changing. Still cool effective solution.
Only 15 lines of code run as root, and they basically just tail kern.log.
Personally, if I was trying to prevent a break-in to a building, I would think it was a great advantage if the attackers had to work hard to even figure out where the doors and windows were.
A more nuanced (and more useful) way of viewing this is one of cost vs reward: How much does it cost an attack to break your thing, and what's the maximum cost that an attacker is lilely to be willing to invest before your thing becomes unappealing. According to this view, obscurity can significantly increase the cost of the attack. It's as simple as that.