Hacker News Reader

Comment by the-dude | Hacker News Reader

I find the outages of the last few days a curious coincidence : Slack, Google, Cloudflare, Azure, now FB.

edit : added Slack after @kache_ mentioned it, added Azure after stevehubertron mentioned it.

Cloudflare's already stated that their outage yesterday was their own mistake: https://blog.cloudflare.com/cloudflare-outage/

Ofcourse. And banks are never hacked or stolen from

Sure but if there were some other weird conspiracy, they would still likely say something like this.

All of my VPS servers have been slammed in the last 5 days with bruteforce SSH attacks. I've never seen anything this aggressive. One of my servers got PWNed, which has never happened to me before. There's definitely some wild, shady shit going on.

I highly recommend installing fail2ban to automatically firewall IPs with consecutive failed attempts, or if possible, disable password authentication altogether and use key auth.

Solid advice. I had fail2ban installed and enabled. SSHD Root/password login turned off and only ssh key had access. My firewall was also airtight, or so I thought. Clearly I mucked up a config or setting somewhere because the odds of someone getting past all that are extremely low. One thing I had not prepared for was IP spoofing which I learned can be prevented with a few net.ipv4.conf tweaks. I also just purchased a static IP from my provider so I can lock down ssh access even further. Here's hoping I never have to deal with this headache again! fingers crossed

What was the vector by which they gained access? I have fail2ban, password login turned off, key access, airtight firewall, etc. and now am worried.

Me too

If they exploited a service or Web app they might have gotten shell access. The chance of gaining access through ssh with fail2ban is extremely unlikely, unless fail2ban was badly configured.

How did you discover the breach, and did you determine the vector? My guess is that it was a pivoted breach from another system on the LAN such as your PC.

I'm still picking up the pieces but from my logs I can see that hundreds of successive login attempts were made from different IPs, effectively circumventing fail2ban with what I can only assume is some form of automated IP spoofing. I'm hoping that strict ipv4 settings and ssh ip range restrictions will mitigate this in the future. I also used this python script to harden my SSH security with better algorithms. https://github.com/arthepsy/ssh-audit

No, you were not seeing spoofed traffic. There are that many compromised machines actively scanning.

It's scary to admit this but you are probably right. The first thing these bots do is use server resources to scan ports and brute force their way into other machines. I don't want to think about how many machines are pwned like this. Very sobering!

This is also perfectly normal for the Internet, yes? If you have a server with an IPv4 address, expect many attempts per day.

> One thing I had not prepared for was IP spoofing which I learned can be prevented with a few net.ipv4.conf tweaks.

Do you have a handy link for more info about that?

`rp_filter`? https://www.slashroot.in/linux-kernel-rpfilter-settings-reve...

You don't generally "get past" your firewall rules and into your box unless you have accounts that are not password protected.

If you really had password logins turned off, you need to identify and isolate how they gained access before you put that box online again. Never "hope" or "cross fingers" that it doesn't happen again. Unless you are an interesting target for some reason, chances are that these attacks are automated and you are running some insecure software somewhere.

Start by taking a snapshot of the machine before you do anything else. Go through the logs. Are there any unwanted processes? How were they started? Are there any unwanted binaries in the filesystem? How were they uploaded? Try to find IP addresses that that be tied to any unwanted login, and see search your logs for any previous occurrences.

Pay special attention to any web-reachable software you have installed.

Did you have password login disabled for all accounts or just the root account?

I don't trust key-only auth; what if I need to access my machine from a new computer I haven't done this with before?

Is there any way to configure SSH to use a custom high-entry password that's different from the user's local password? My local password is something reasonable for me to type regularly (e.g. for sudo prompts), but I'd love to have a super long password just for SSH that I have to copy from my password manager each time.

This could be done simply by creating a new user with your super long password. Lock down ssh so you can only log in as this user, and let them `sudo su` into your normal working user.

I mean, yeah, but I don't want to do that. Partially because I don't want the friction, and partially because that won't work with any other tools that tunnel over ssh (e.g. sftp).

There will be friction, but sftp and port tunneling - which are my most used fwatures besides plain ssh - should be possible?

I mean either you sftp to a shared folder that can be accessed from your regular user as well or you use a staging area with a cron job (or you load/unload the staging area manually.)

If I'm sftp'ing to my server it's because I want to access my files. Not a special shared folder that I then have to separately ssh in, su to the real user, and move into place.

I'm not deploying a website so a staging area isn't applicable. This is just a VPS that I use for various purposes.

If your ssh user has the right priveleges, you can read and write your real users files as the ssh user just fine. I do get that this isn't ideal though.

There are definitely cases where you might need to expose a an SSH setup where passwords are allowed to the World, but there is usually little reason to allow anyone to log in directly as root.

Set a long password for the user you log in as (correct horse battery staple-style passwords are perfect for such things), and make sure to put SSH on an alternate port to keep the more basic bots away and thereby reduce the noise.

Having to type a long password for sudo promts is a bit of a pain, but that trade-off is worth it from a security perspective.

If this is a scenario that happens often, you probably should invest in a portable method to hold your ssh keys, such as a smart card. I have only used YubiKey for this purpose, but I'm sure others, like the Nitrokey, work similarly.

"a custom high-entry password that's different from the user's local password?"

You mean an RSA private key? (no really, that's exactly what it is... you can put your private key in your password manager and copy it to your computer)

I can manually type in a 30-letter password that I can see from my password manager on my phone. I'm pretty sure I'm not going to be manually typing in an RSA private key stored on my phone.

+1 for disabling password auth. On the other hand I find fail2ban pretty useless these days. Most attacks I see seem to come from botnets where you only get a few requests from each IP.

You can also setup a VPN and limit all port 22 activity to your VPNs IP.

Yep. I just paid my ISP for a static IP address to accomplish this exact thing.

I know it's just an obscurity measure but step one is surely don't expose SSH on 22? Dropped my (home) attacks/scans to zero from hundreds per day.

You could do that but if you become a target due to unexpected viral growth then it's useless.

It's more "stops you appearing on Shodan" level. But surely if you can reduce the traffic then you are more likely to notice proper attacks.

Shodan is amazing. Creepy too. I used to spend hours trawling video feeds on there. Fascinating.

You should disable password auth entirely and whitelist the IPs you connect from in your firewall. If you need to conenct from a large range of possible IPs, use a bastion host with 2FA and restricted to IP blocks from countries you actually connect from.

Several european banks are also having issues with their login sites and online services. Backends are working though. I know, it's far fetched to throw them in the same bucket, and I won't, but it feels like there is happening some kind of reorganization on the net, which in the worst case would be some kind of global state-sponsored MITM-layer getting set up, maybe to prepare for new techs like 5G and TLS1.3.

My metaphorical tinfoil hat tells me that all the sudden security meetings yesterday (Putin, European Security Council, US VP Pence) were about this.

If I were going to buy into any theory other than coincidence my bet would be on an emergency patch situation where we're seeing the rushed deployment.

But it's probably just a coincidence.

That is not terribly far fetched. Cisco powers a lot of the core and has had nasty zero days before.

And on top of that highly classified Russian submarine allegedly used to tap on undersea cables has sunk on Monday.

One of the biggest banks in Brazil (state owned) is currently having the same problem, bb.com.br

I agree that the probability of these occurring so close together is very unlikely.

Separately, this is a huge blow for Instagram/Facebook since this is one of the busiest social media sharing weeks in the US. I bet a lot of their engineers are answering pagers on vacation.

If you have your work with you, it isn't a vacation.

If it's emergencies only and you get compensated handsomely many people will feel it's worth it. Of course reality often looks different.

depends on how you value your vacation time with family/friends/to yourself.

In my experience, people jostle to take part in a rotating on-call roster where they are sufficiently compensated, in my case $1700/week, to occasionally answer one or two calls.

A lot of my vacation destinations don't even have cellular signal. Taking a laptop is often not an option on some trips either.

Add Azure to the list. Its routing was down for about 2 hours yesterday.

Actually, I started wondering the last few minutes : what if the next worldwar would start on the internet? Or is this a common assumption already?

I had not connected Huawei to my conspiracy theory yet, but just did.

People who make defense policy have been assuming the answer to that question was "yes" for about 20 years now. It's sometimes described as "cyber precedes kinetic."

https://www.rand.org/pubs/monographs/MG1215.html

Thank you. So we are basically waiting for a Twitter outage.

If Dropbox goes down, then something serious may be going on.

I think Twitter DMs are having issues too...

Ok, so unfriendly sovereign nations are trying to “hurt” us by making social media inaccessible? I’m half joking here, but perhaps we should all be out, about, and enjoying Independence Day in person rather than through our phone screens.

(Yes I realize WhatsApp is a different segment, an important way people communicate.)

All three apps are huge for communication. It doesn't seem far fetched that a nation would test its capabilities taking down different forms of communication.

You might love Neal Stephenson's latest book "Fall, or Dodge In Hell"

It's not quite about a world war on the Internet, but it is about what could happen if the Internet suddenly stops working somewhere.

Nitpick, but it's:

Fall; or, Dodge in Hell

People would start by attacking telecoms much like they used to start with rail and harbours.

If I'm not mistaken virtually all of the ones in the list above were caused by networking problems. Given how much Chinese parts are in those interfaces, it does make you wonder.

It could also be just a really strange coincidence.

So, people like to slam on GCP but not Azure. Didn’t make front page of HN but GCP wasn’t even an outage just increased latencies due to rerouting and still make front HN?

So the options are:

1) a secret cyber-war is going on...

2) Some entity is installing new spying slurps

3) An (critical) IX that we werent aware of is having major issues... and we dont know who to blame on that one...

4) People love to find patterns where none exist

This is a fallacy all humans experience, however, what is the probability specifically that all these problems are unrelated? At what probability is it mathematically impossible that these problems are unrelated?

AFAF

The answer to your last question is 0.

"Impossible" means "impossible", not "unbelievably improbable".

That's even more true when you ask about "mathematically impossible", as it reinforces the idea of formal logic being the relevant domain, where precise meaning of words is fundamental.

If you adjust your question to be "at what probability is it unreasonable to claim these problems are unrelated?", then the answer is subjective - different people have different standards for reasonability.

I think we'd need mountains more data than we have about the incidents to compute a meaningful probability, anyway.

is that a patern ?

5) patterns exist and people who have been online for decades can see them.

(Your HN account is 4 months old. Mine is ten years old)

Honestly, I think it's much simpler than that -- the last decade has been a general trend of building more centralized, less resilient systems. The global internet is in an inherently unstable state because of this (i.e. when problems occur, the internet is no longer able to route around them because the problems are internal to large organizations which both control huge portions of the internet and which aren't internally incentivized to separate components -- see e.g. the AWS status images living on S3). These outages are simply more and more frequent as the system's instability increases. And they will continue to become more frequent as time goes on, until organizations realize that they can't control their own uptime unless they run their own infrastructure, at which point the pendulum will start swinging back the other way. Rinse, repeat.

0day patch gone wrong? We've seen before that these companies work together to patch issues behind the scenes before announcing them.

1 & 2: Could it be? https://futurism.com/russian-sub-fire-internet-cables

Uss jimmy carter

4) it's summertime in the predominantly online part of the world and general human activity is at a peak

Hmm...

I like this theory...

Its innocent but id like to see data behind it.

Empire BlueCross/BlueShield's site is down. Not quite in the same category but sucks if I want to file claims.

https://www.empireblue.com/

+ A large unnamed government contractor that has also been having some rare network outages throughout the week.

HN still seems to be working, perhaps they should consider hosting their own cloud?

HN is probably running on a LispMachine next to pg's desk. I wanted to say "under his desk", but you know, LispMachines...

Seems its still a single server. I think PG is off on sabbatical still though. https://news.ycombinator.com/item?id=18184749

Might be a desire to release new versions ahead of July 4th holiday.

"When it rains,it pours"

In my experience this is so true! The opposite applies as well,for some reason everything behaves quietly all at once and I turn up every leav and look under every carpet for a "problem" with the monitoring or something worse.

Probability is funny, for example I don't think anyone has explained exactly why numbers that start with '1' are so common. I think it's just geometric symmetry if complex event chains.

https://en.wikipedia.org/wiki/Benford's_law

Thank you,i think this has to do with higher order numbers' symmetry with each other

There was a huge Comcast outage in the bay area last week as well, although most people probably didn't notice since it happened at night.

Twitter notifications and DMs were also down today.

Heard some rumors that China might have something to do to this, related to the Hong Kong protests, but they're just rumors.

where are you seeing the rumors, if i may ask? i'm not interested in propagating mistruths, but i'm interested in digging deeper.

Not the poster you replied to, but searching OSINT seems to cover this. Open source intel - though a lot of that is actually on Twitter.

Whatsapp is having problem delivering downloads now (at least where I live)

Could it be a physical infrastructure problem? There are occasional issues with fiber cuts that cause outages. I'd imagine many companies rely on such a critical piece of infrastructure.

Apple Music seems to be having issues too

If I would take a wild guess : it's the heat waves. They drain people's concentration...

Why curious? Is there anything to make you believe there is foul play?

Curious because those are big services people expect to see up 99% of the time. To have them all fail in the same 48h is a curious coincidence.

99% is actually a lot below SLA for these services. It means the services are down for 3 days 15 hours in an year.

Why curious?

People naturally find coincidences curious. It's how we find causality.

Often even when there is none to be found.

Jeff Bezos casted a spell to crush his opponents \s

And today, eight days later, Twitter.

Imagine if Google is next

Google had network issues this morning in us region

They were down yesterday

Reply on news.ycombinator.com