DigitalOcean Root Vulnerability in the Wild
badassrockstartech.com
badassrockstartech.com
That is a bit ridiculous that they retained your password. Still though, logging in after rebuilding and configuring your system would have been a sane move.
Probably not strictly related (it would seem the attack in question was via a compromised (on whoever's end) ssh password), but in any case good practice: always regenerate ssh keys after installation from a base image. Some OS images may contain pregenerated keypairs (private key is usually at ~/.ssh/id_rsa ), which should always be regenerated. (This bit is most probably not actually related to the issue at hand, but IAC should not be forgotten.)
(No affiliation with DO here, just hate to see any company unfairly attacked...)
Edit: Disclosure, I met one of the co-founders at SXSW. Seem like cool dudes.
Yes, I know my setup was very flawed. It was a test machine and in the midst of having automated config scripts written for it which were being tested. This is why the machine was vulnerable to the attack, but this does not diminish the importance of the underlying compromise it exposes in DigitialOcean's setup.
The point of the article is that someone is able to gain access to DigitalOcean password reset emails or a database of root passwords which shouldn't exist, but seems to given that they set your root password back to a previously reset value after you rebuild your server from the base OS image.
"Your root password will be emailed to you"
-- start --
Me
Just seen this at the bottom of the "Create a droplet" page. You're kidding right?
Them
The root password is sent via email because it is the easiest and fastest way to get a user online and running a virtual server.
We strongly recommend updating the root password after you login for the first time.
We also have SSH keys support so you can add your SSH key to the server during creation in which case no email is sent and instead the SSH keys are added under the root user for more secure access.
Thanks,
Me
Just added an SSH key and you're quite right, I retract my blunt reaction.
I must say however that I still think emailing them is a fairly terrible idea and I'm surprised your not worried about being found liable for a subsequent server hack.
That aside, thank you for your swift response and for pointing out I can use my SSH key. I look forward to using DO more
-- end --
N.B. I actually received their response twice from different agents suggesting it was a canned response
Frankly I don't even think passwords should be an option, as on AWS (not that they're perfect)
Heuristics like "keys are better than passwords" or "don't email passwords" are good heuristics, but they shouldn't become absolute rules.
Onerous security policies can make things worse as people work around them behind your back. You can require certificates but there's nothing you can do to make sure that people don't put their keys in their dropbox for "safety" because they don't want to get yelled at for losing their keys when a machine crashes.
People shop for VPSs partially based on convenience of management. If you make your system secure but inconvenient, you can lose the people who would would most benefit from a "good enough to stop the automated SSH bot" policy.
Those are the first few things you should be checking when setting up a new server (disable password logins and only allow keys).
shocker.
(It's very likely worse to do your own unless you have the resources to do it right. Hosting companies have scale. Not defending DigitalOcean here, just challenging your assertion.)
It took 5 support letters to get my account unblocked, and the solution was to block SMTP ports...
I didn't have anything installed !!! Clear Ubuntu image
Once a system is compromised by a root exploit what makes you think that any information that this system is giving is true?
While it may be likely seen the circumstances it is by no way certain. For all we know it may have been a bruteforce attempt which, once in, got disguised as a known-root-password attack.
As long as people are going to think that a compromised system is actually giving true information about what happened we'll be in big trouble.
Or OP tells us that SSH login and SSH login attemps are logged automatically on another server which hasn't been compromised and then it's a different story...
What if a botnet is used to do the brute-force, does fail2ban lock everybody out, effectively making fail2ban a tool that can be used for Denial of Service?
All I'd have to do now is to hammer your system with SSH attempts from my botnet and you can't log in anymore...
Therefore it must take IPs into account. Therefore it's not protection against brute-force password guessing from a botnet.
Once you're in that fairly secure state, fail2ban is completely pointless and serves only to reduce noise in your log. Which you should stop caring about. I'm serious. fail2ban is a complete waste of resources and will peg a core tailing your SSH log because it's a resource-intensive piece of software that serves no valuable purpose. If you're worried about the resources of writing a log, redirect the log off the box and prevent your local logger from writing to disk -- which you should be doing for security reasons anyway, as discussed in another comment elsewhere in this thread (how do you trust a box's logs once compromised?).
While you're at it, don't move SSH to a separate port. There are interactivity QoS defaults in most networks that expect SSH to be on 22, and the clever bots will find your "cleverly hidden" daemon anyway. And wait until I tell you that in the event of your "cleverly hidden" port 2222 SSH daemon crashing, one of your local users can put up a trojan SSH daemon in its place without your permission (which is why SSH is on 22 in the first place).
Just deal with the log noise. We all do on edge boxes exposed to the Internet.
But, yes. There's an iptables module called ipt_recent[1] that will do what you're after. It's been built-in for ages and you don't need to install it.
You can't use it by itself, though. You will have to design a full iptables ruleset.
echo " (eth0 - SSH rate check)"
iptables -N SSH_CHECK
iptables -A SSH_CHECK -m recent --set --name SSH
iptables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 3 --name SSH -j DROPAssuming that's true, now guess what happens when you do this during a crisis:
for i in backup-file-{0..999}; do
scp $i that-machine:/var/db/
done
I really wish we could reverse the trend of fiddling with iptables when it comes to SSH. Everybody does it and nobody thinks about the ramifications.
The global limit on scp is fine; this is an internet-facing server with minimum user data for which I have a cold spare - there's nothing to back up.
You're coming over like you're the only person in the world who knows what they're doing; might want to rethink that.
Every HOWTO I see that people copy and paste considers every SSH connection as ammunition for the ruleset. So if you connect to a machine however many times -- and, scping a couple files opens new connections unless you have a clever muxing config on the client side -- boom, the firewall will intervene based on most of those copy-and-paste blog entries.
Smart admin edict: do not mess with my ability to SSH in under any circumstances. I've been in this situation, if you can't tell from the specificity. ("Why is the edge not responding to SSH any more? Oh, crap, we're flagged by the firewall and have to wait ... I forget how long.")
> It's almost always a good idea to change ssh to a high-port
[citation needed]
> QoS is irrelevant to most
You sure? QoS is even in consumer routers these days. I'm speaking from experience, and have shown a 20% drop in scp performance by moving away from 22 on a consumer Netgear. This point really isn't worth caring about, though, but...
> bots do not scan !=22
I run a honeypot on 2222 with a well-known address. It's been scanned by 47 bots (out of 1,449 against 22) in the last 48 hours. They're out there, because it's not exactly a secret that administrators move SSH. APTs will scan every single port looking for OpenSSH, because it announces itself in the opening conversation. Granted, they are a smaller figure, but your assertion does not hold water.
You are also building upon obscurity. Ports are just endpoints. If you shuffle five feet to the left, you're still very likely standing in the same room. Your system is secure with SSH on 22. Period.
> if an attacker can launch daemons on your server then you've already lost anyway
You didn't read what I wrote carefully. I did not say attacker.
Ports under 1024 are reserved for root. Unless you are UID 0, you cannot bind to a port below 1024. That's why SSH is, by default, on 22. That is a service that only root should be able to start.
If you move it to 2222, say, inside a company of a bunch of employees, one day I might get clever after your sshd crashes or I somehow coerce it to crash (and there are ways). Now there is nothing listening on 2222, but I have a physical console, and I launch my own trojan sshd on 2222 (totally legal, because I can bind to 2222) and capture passwords from everyone that connects. Now I have passwords from all of my fellow employees, and I can start trying other systems.
That party is not an attacker. He is an employee that you gave an account. And do you have monitoring checking that the sshd listening on 2222 is running as root? Didn't think so.
Ports above 1024 subvert the security model of Unix, and should not be used for a system-critical service. Ever. If you are going to move it even against this advice, do not go above 1023.
That being said, I'd like you to provide one good security reason to move SSH to a separate port. To be honest, this whole "move SSH to a high port" is a complete and utter lie started by someone and parroted by every administrator who heard it from another guy, and it is rooted in absolutely nothing. It's not my job, as you've demanded, to justify leaving SSH at 22. It's your job to justify moving it.
The rest of your comment (faith in privileged ports, hollywood attack) doesn't lend much credibility to your advice.
You shouldn't depend on it, but being a bit out of the way means you have less people rattling your doorknob.