DigitalOcean block storage is down

Would be great if statuspage.io has a button when pushed publishes message similar to your suggestion.

This does seem to indicate a notable lack of isolation for the blast radius between DO datacenters. Would be interesting to see the post mortem.

Probably the old "one command ran on everything"

It could be DNS. Azure has had an all-region failure due to a single DNS provider outage. It was possible that same DNS provider's outage was also causing problems for GCE and AWS at the same time.

https://news.ycombinator.com/item?id=19812919

Bad puppet/ansible/etc commit is the most probably explanation.

dee ennn esss

A bug that has no obvious side effects that only became visible once all data centers were upgraded?

Happens. Statistics are hard.

I work for a web hosting company in Texas, and this is ridiculously common. Any public IP with any public service at all will be poked, prodded, and generally made uncomfortable by every bot and crawler you can think of, trying common password combinations and scanning for common vulnerabilities in popular software. This catches so many of our customers by surprise, who tend to mistakenly believe they're being targeted in some kind of attack. Generally they're not, unless they're running something vulnerable and one of the bots noticed.

Fail2ban is great to at least stem the tide. It's good at slowing down SSH brute forcing, and can be set up to throttle poorly behaved scrapers so your site isn't getting hammered constantly. If you can deal with the inconvenience, it's even better to put services that don't need to be truly public behind an IP whitelist. That stops the vast majority of malicious traffic, most of which is going after the low hanging fruit anyway.

Otherwise, it's kinda just a fact of life. With the good traffic also comes the bad.

> Is it normal for a very private vps to receive thousands of ssh attempts per hour?

Yes. The thing is about the IPv4 space is that it’s really not that big (3,706,452,992 public addresses) so it’s pretty trivial to poke every single one esp if you fine tune your port list.

The most common advice is to hide your private services. Instead of using port 22 for ssh use 23231 instead. It’s a little more annoying but you can also use port knocking. So to open port 22 (or what ever port you like) first you got to poke port 23123 then 7654 then port 39212 within a short period of time then the port knocking software will open up port 22. (Or a combo of both change default port and port knocking)

It won’t stop people “ringing the door bell” to see if anyone is home, but it will help with the then trying to brute force pass the login prompt.

I work for a hosting company and this is totally normal. Digital Ocean and other VPS providers IP ranges are specifically targeted as many amateurs running servers there.

If you've disabled password logins then just don't worry about it. fail2ban is overkill you can rate limit with firewalld or iptables withou needing extra tools.

It is "normal". Even my home fix IP gets it without any service running on it other than ssh.

I had the same experience on DO as well as a few other providers.

Changing the ssh port to something in the 50_000 range drastically reduced the number of attempts and left my logs much cleaner :).

   Is it normal for a very private vps to receive thousands of ssh attempts per hour?

Well, I haven't bothered looking in a long time. But, back when I first got a cable modem back in the late 90's the malicious access attempts filled up my hard drive in just a couple of weeks. I don't remember the size of the HD, but I can only imagine the situation has gotten much, much worse since then.

Yup this is normal, when I can't change the SSH port (e.g for compatibility) then I switch f2b to permanently ban, which should reduce any incurred load by black-holing instead of attempting authentication as the list grows more comprehensive over time (this wont affect other services e.g apache, in-case a user is unwittingly part of a botnet).

Off topic reply, you should monitor the amount of malicious traffic coming from DO networks too. (I did for a few customers at different isps and its insane)

I took a look at the flags on these stories and am pretty sure they're from users who are tired of "X is down" submissions, which tend to get posted a lot and often to be a little on the trivial side.

However, since several HN users are expressing that this issue is genuinely affecting them, I've turned off flags on the OP about this and merged the comments here.

I reported it and they were like "what? oh..."

Then the status page changed and as things got worse, the dashboard page got an announcement as well.

Yes, DO uses Ceph: https://blog.digitalocean.com/why-we-chose-ceph-to-build-blo...

'The Cloud' is _a_ backup. Just don't let it be your only backup!

that would be pretty cool but to have that, you need a high-network-latency solution, i.e., pretty much cold back-up. For some time I thought it's pretty last century option but having been experimenting for some time now, it's the option with lowest impact on system performance. More importantly, it's reasonably resilient.

Yeah, the myth that "just use aws" to have 99, 9999999 percent uptime is coming to an end...

That wording kinda ticked me off because our volume was completely inaccessible. Rebooting did not mount it at all.

BitBucket definitely doesn't use DO.

Since you're trying to "delete file with lifecycles", I'm quite sure your problem is with their object storage (called Spaces), and not their block storage.

great idea - going to give it a whirl this week.

i'm curious about the slack integration. can you provide some more info on what that looks like? eg. just a message in real-time when it goes down? a daily message of statuses? etc. Any sort of customization w/ it?

I currently use a soup of zapier zaps to take care of this problem.

Last night I was testing DO managed Kubernetes cluster with persistent volume claim and the volume took 15 minutes to reattach after the pod is rescheduled to another host. I thought it was just some weird hiccup and went to bed.

The incident report indicated the problem started 4 hours ago (around 9pm GMT) but I was having problem around 4pm. It's definitely not a 2-hour incident.