The Windows servers that have been fueling DDoSes for months | Hacker News Reader

The Windows servers that have been fueling DDoSes for months

arstechnica.com

112 pointskungfudoi86 comments

That article is completely wrong. Misconfigured servers that amplify traffic are not a problem at all. Blacklisting all of them would take no morethan several hours.

The problem is that there is no protocol for rejecting traffic. That's why DDOS is so easy and successful. If you are a website you cannot send a message to uplink saying "I don't want traffic from this network". So small and large ISPs do nothing to help protect victims and make it easy for criminals to extort money.

This sort of thing is being done as far as I know, but I'm not aware how big a customer you have to be for your ISP to care about such notices. Perhaps so large that you've nearly got enough capacity to handle the traffic yourself anyway.

It also creates a nice DoS vector: if I want to not take an exam today, I'll just do a little ping flood from the school network and get banned for a while. Similar to account locking on 3 invalid attempts, except now you can block everyone at once as soon as you're in the network or on the same ISP.

As a small customer I’ve experienced them doing the complete opposite. I’ve had a 1u collocated at one center or another for the last ~15 years and occasionally we would be the target of a DDoS. The data center would just blackhole us until things calmed down. It’s better than a surprise bandwidth bill but certainly not great.

FWIW, that's what happens for large customers too, from my past experience. Someone aims the DDoS cannon at one of our webservers [1], and if the traffic is above the threshold, that server's IP gets blackholed for a while. As a large customer, you do get to have some input into the thresholds, and how long the blackholes are, and to definitely have traffic blackholed rather than put through some crazy filtering apparatus that adds so much latency it'd be better to drop everything.

OTOH, we always had at least 4 webservers, so if just one got blackholed, no big deal, update DNS so most of the real users make it to the available servers, most of the abusers will keep hitting the blackholed server and not move on. If your abusers aren't run of the mill script kiddies using a rent-a-ddos service, it's more difficult.

[1] it's almost always the webservers, not the servers that do real work

How large is a large customer here? Paying the hosting company monthly bills on the order of 1k, 10k, 100k? I expected this to only affect small and medium businesses, so I'm genuinely curious what qualifies as large but is still affected. (My security consultancy work only tangentially touches upon DDoS, so I know a thing or two but am not actually in this business.)

My experience is a bit old at this point, and I don't remember quite where exactly we were when we got to have a meaningful conversation on getting the blackhole settings set to reasonable values [1], but our bills were at the high end of your range, and we were a top 5 customer at some point.

For us though, getting one server blackholed every so often wasn't that big of a deal, so as long as everything was managed reasonably, and the ticketing was timely so we didn't spend a lot of time trying to figure out why server X wasn't working when it was just a blackhole from DDoS. A clever abuser could have caused a lot more trouble, but clever people tend to find more productive things to do.

[1] don't blackhole unless the traffic rate exceeds line rate (which topped out at 2x 10G), retest after 1 hour instead of 24 hours for the first retest, etc.

Ah yes, I know those stories as well. I found that surprising also, but it seems to stem from the botnet era where there were thousands upon thousands of originating networks instead of some big (DNS) servers. The practice is still in place though, especially at some cheap hosting providers.

So basically the data center took the side of the criminals.

It is sort of a common practice to get rid of "problem" customers too.

We ran a pretty large Counterstrike gaming community with public servers, and did some very small scale paid server hosting on the side. It's unfortunate that anyone would consider us "problem" customers because DDoS attacks are so easily performed by script kiddies. It can happen to anyone hosting public resources that aren't easily filtered through no fault of their own.

I totally understand what you're saying though.

Because that makes incoming traffic not to come to them at all. Traffic costs money and usually much more than a single 1U colo cost.

BTW nice username

No doubt, we weren't able to pony up the cost difference to cover the bandwidth in cases like this so I get why this was their move.

You have a nice username as well!

> but I'm not aware how big a customer you have to be for your ISP to care about such notices

So basically ISP don't want to do anything to prevent misuse of their infrastructure because it is more profitable this way.

> It also creates a nice DoS vector: if I want to not take an exam today, I'll just do a little ping flood from the school network and get banned for a while.

The idea is that you can only block traffic going from school network to your site, but not any other traffic from that network.

The argument is that the servers aren't a problem - the problem is that we don't have a protocol to block them. Of course, we wouldn't need a protocol to block them if they weren't a problem. I'm very confused by this argument.

"X isn't the problem, Y is" is the overconfident way of saying "Y is a better solution to the problem"

Misconfigured servers that can amplify are only one way to DDoS attack. Even if this wasn't a problem you would still have all the other ways to DDoS, which you could handle easier if you could actually block the traffic.

So yes, we would still need that protocol.

You mean something like RFC5575? That actually exists and is in use.

There are protocols in development at the NIST such as Named Data Networking[1] that implement such a scheme through flow balance and interests.

[1] https://en.wikipedia.org/wiki/Named_data_networking

and the public does what for the next 20 years before that reaches beta stage?

Also, hosting vendors that allow customers to spoof source IPs. BCP38 would nicely fix all of that and most reputable hosting providers have it, but all it takes is a few to not use it to give this possibility.

You're saying the solution is to have every website in existence find all these IPs and then request that their ISP block them? That sounds like a lot of work.

Sure it's a good idea to allow requesting the blocks. But changing the reflection sources themselves so they can't be used for reflection anymore is also a good idea.

So, government will be able to silent anybody anywhere in the Internet just by using standard protocol. A Great World Wide Firewall.

They will be able to do it only for their citizens and they can do it already.

Misconfigured servers are the basis of multiple problems not just DDoS attacks.

A new protocol only knocks off one head of the hydra, but does not solve the real problem.

The article is correct and MS is at fault here (also per the comment there)

"The problem is that the non-exisiting solution doesn't exist" - ok then

You're wrong. Swap "spam" for "DDoS" in the following.

https://craphound.com/spamsolutions.txt

BGP blackholeing is used routinely.

So now uplinks are going to maintain lists of networks you don’t want traffic from. How do you imagine this is going to work with IPv6? It requires unbounded amounts of memory and processing power.

Wait... so there are thousands of Windows Domain Controllers just... "on" the Internet!? What are these people thinking?

Everybody is on "internet" now. These people are not thinking. Just like it is normal for your average phone user to have wifi, cellular data, bluetooth and location always turned on, so it is normal for these admins to have their DC on the internet.

I've managed Windows servers since the NT4 era, and it would never occur to me to make a public-routable domain controller. If the requirement came up, I would go to the n-th degree to make it "Internet safe", but how could you guarantee something like that!? It has dozens of protocols enabled, most of which were designed for private networks in the 1990s!

Microsoft has just recently made SMB safe for the Internet (Azure File Shares), but that's just one of many protocols on a Domain Controller...

i kind of share your sentiment about putting windows machines on the internet, but otoh ther are a lot of companies doing it and azure ad is almost dominant.

> Microsoft has just recently made SMB safe for the Internet

cifs anyone?

You're mixing up your products.

Azure Active Directory has almost nothing in common with Active Directory, other than the name. Amongst other things, it uses HTTPS almost exclusively.

CIFS is an older protocol that Linux admins use incorrectly to refer to SMB, which is the file sharing protocol used by Windows.

MS called SMB 1.0 CIFS. <https://learn.microsoft.com/en-us/previous-versions/windows/...> . So it seems windows admins are using the old monikker.

SMB 1.0 isn't even included in Windows any more.

Is Linux UNIX?

And yet people don't consider AD to be an dangerous legacy technology on it's own.

The enterprise world have a huge problem with it's addiction to legacy Microsoft technology that Microsoft is both aware of but also unable to fully address as their modern stuff just aren't all that competitive without the legacy compatibility. And the "domain controllers" it's at the very core of this problem.

In the unix ecosystem most of the old vulnerable legacy(which was never as bad as windowsNT) is actually dying out but for some reason most of the traditional wintel ecosystem seems to be partying like it's still 1999.

Before I made the career switch to pure GNU/Linux, back in my MSP startup days, I made good money ripping out horribly run ADs leftover from the great sysadmin layoffs of the late 2000s. Samba 4 came later and was a godsend. AD and MS in general has stockholm syndromed many people.

Really and truly the problem is executives not seeing the value in hiring for their actual IT problem space. Too many companies, especially ones that are more physical and have people who like to proudly claim computer ignorance, refuse to admit that with advances in technology that IT is an integral and vital part of the business.

Instead they don't even hire for positions they really need, or if they do, they hire for peanuts and get subpar gui-ninjas... and that's how you end up with a bunch of AD servers on the internet.

If your DC isn't public-routable how do you get updates?

Do you mean the clients connected ? Most DC's I know are firewalled from the internet basically on an internal company network. If you want to get group policy updates and suchlike you connect to the internal company network via VPN.

If you are talking about how the DC itself gets updates then it has outbound access normally via a proxy server or something like that.

You wouldn't put the DC on the internet that would be really insane.

But how then does people working from home logon to their AD joined computers?

if you cannot expose the AD server to the public internet, can you then actually use windows in remote first organization?

Those servers have been exposed to the internet because if they weren't the organizations using them would be forced to solve the chicken and egg problem of how to authenticate the users against an authentication server they cannot access without authentication.

You can use Okta, AAD, etc to front your AD with OAuth2 and secure your VPN with that.

> But how then does people working from home logon to their AD joined computers?

I work in this exact type of environment. Your laptop is joined to the domain in office before being sent to you. You can of course logon to the laptop even though it cannot connect to the domain. You just have to connect to the VPN and domain periodically to get gp updates if there is some mandatory software it'll install when you connect. So you connect to the VPN after you logon locally.

You also have to connect to the domain to change your password.

> Those servers have been exposed to the internet because if they weren't the organizations using them would be forced to solve the chicken and egg problem of how to authenticate the users against an authentication server they cannot access without authentication.

No you can logon to a computer with your domain credentials even when it's not connected to the domain.

Active Directory Federation Services: https://learn.microsoft.com/en-us/windows-server/identity/ad...

The internet facing deployment means you have to set up ADFS Web Application Proxy in DMZ that is NOT domain joined and can communicate with domain-joined ADFS service.

ADFS allows plenty of stuff, including 2FA, for example for trusted devices only, which has been registered with ADFS and uses certificate based auth. It allows SSO with online services like Azure, Google Workspace and more.

Edit: Ahh, sorry, the story is about not logging into web apps, but domain joined computer. Credentials are cached on user laptop, so you can login without network (you can prevent credential caching if you will. You should for domain admin accounts). Or you setup always on VPN (DirectAccess or wireguard) so that you always have the connectivity.

One of the servers inside your domain should be set up as the WSUS server. Through group policy, you set all computers inside your domain to get updates only from that server and only approved updates.

I have not done network administration in a very long time, so my suspicion is that many of these domain controllers were set up by not very experienced people with not very intelligent bosses thinking that "it was done once, therefore it is done forever".

> What are these people thinking?

They don't. MOre so - they don't even know what they are doing.

NB there are people on the other side of the spectrum - I've met enough Linux fanboys who insisted what they don't need a firewall running on their public machines, because... Linux is safe.

Which is true? Most Linux distros don't open any ports by default on a fresh install. if you don't want some port to be reachable, don't install the software that opens it?

Obviously in a more involved (enterprise) setup a firewall will make sense for several reasons, but on a small server you rent for yourself to run an httpd and let's say an ircd plus ssh, what exactly do you think a firewall does for you?

It's easy to disable firewalld and groupinstall anything. Stupid things can be done.

> don't install the software that opens it?

> but on a small server you rent for yourself

Bwahaha!

But what if I WANT to install software that opens it? Like MongoDB for example? What could be wrong if I just install some MongoDB to run $softwarename, right? Nothing wrong could happen, yes?

Read my comment at [0], I don't want to repeat it here.

[0] https://news.ycombinator.com/item?id=33411185

> I've met enough Linux fanboys who insisted what they don't need a firewall running on their public machines, because... Linux is safe.

Why would it need a firewall running?

The same reason you wear helmet on a bicycle or a hard cover hat on a construction site. It is an addition layer of protection (and with firewall you don't even any discomfort of wearing a helmet) which can protect you if something goes wrong.

Today you setup you shiny new host and you know exactly what is running on it and what ports are open. Some months/years later?

Yes, a single purpose mail server can get by with tcp/22 and tcp/25 for all it's useful life but something more complex?

Or when you don't even know for sure what are defaults are and you 'move fast and break things'? Should I remind you about MongoDB fiasco? [0] Are you sure all your 'internal' services (DBs and whatever) are bound to localhost and not to 0.0.0.0?

Running a firewall in the default drop/whitelist mode (at least for the inbound traffic) protects you against your own mistakes.

Add to this what while you can be an exceptional localhost admin, as soon as you have more than 5 hosts and/or work with other people you can never be sure what every host is properly configured, secured, don't have anything unnecessary running.

[0] https://krebsonsecurity.com/2017/01/extortionists-wipe-thous...

>> If installed on a server with the default settings, for example, MongoDB allows anyone to browse the databases, download them, or even write over them and delete them.

So about the only thing you can give an example of is an application that no-one should be running on a server exposed to the Internet, that's dangerously broken by default?

So about the only thing you can object to is what I didn't provide you a comprehensive list of a networked software with an unsafe defaults, now or ever existed?

Edit: with a 'deemed safe enough by Gordonjcp@HN' column of course.

It's also unsafe to drive around with your eyes shut, but any reasonably competent person would know not to do that.

Your argument is "software might have unsafe defaults, so you should rely on a piece of software that might have unsafe defaults".

> Your argument

My argument is having an additional layer of protection just in case. Do you wear a helmet only on the days you fall off from your bicycle?

> you should rely on a piece of software that might have unsafe defaults

Well, if you just took the helmet with you but never bothered to wear it...

Your analogy breaks down. It would be more accurate to say "Do you wear a helmet only on the days you plan to intentionally launch yourself off the bike into a rock?"

If you cannot install a server without ensuring you're running what you expect to be running, you've no business running it at all.

Tell that to all folks who have no idea but eager to have a private server/make business/whatever. It looks like you are one of those fanboys who regard their skill as a something sacrosanct to which no plebs should be allowed to even know.

Not in the slightest.

It's a skill that everyone should know, if they want to run servers. It's a skill you can acquire by reading some very simple instructions.

Imagine if bridges were designed and built by the fuckwits that created (or indeed use) MongoDB.

I don't wear a helmet so I can ride my bike drunk with eyes closed.

The other way in which the analogy is flawed is that I might very well consider myself a perfect cyclist, but that still doesn't protect me from that drunk driver in a SUV shooting out a side alley where I have no chance to react.

Your analogy still doesn't work.

A firewall does nothing useful if there's nothing to firewall.

Yes? That's exactly what I'm saying.

Sometimes you want LDAP open to other places for auth and directory sync, ideally it should be IP restricted but some people don't do that for various reasons.

> What are these people thinking?

Probably something like "hey, my nephew is a computer nerd, why not hire him instead of $EXPENSIVE_CONSULTING_COMPANY?"

Followed a couple months by: "Why is our internet so slow?" and "why aren't out customers getting our emails?"

It'd be sensible to the ISP to just disconnect these from the network, particularly if they can't get their shit together after a warning.

I keep thinking the same. It's the best option short of just targeting it against themselves and causing just enough nuisance that they either firewall, upgrade, or correctly configure these ancient protocols. For example for the protocol that this article is about: "Windows 2000 Server requires UDP" https://wiki.wireshark.org/MS-CLDAP. Anything newer than Win2k means you can already turn it off.

To me it's crazy that a religious organization apparently has a 17gbits+ connection to the internet. Admittedly I'm not an expert and only know of one that I'd consider large enough here, but yeah... Second question is why their ISP never had any alarm bells going off with such a traffic pattern.

TL;DR of Ars: Connectionless LDAP

Actual article that Ars writes about: https://blog.lumen.com/cldap-reflectors-on-the-rise-despite-...

Actual news (less vague title than Ars'): the number of vulnerable systems has increased ("from the 7K range to over 12K" "over the last 12 months"). Only 15% stays online longer than a year, though, so it seems that most admins catch on after a while. "[T]he Connectionless Lightweight Directory Access Protocol (CLDAP) [has a] Bandwidth Amplification Factor (BAF) of 56 to 70x".

Actionable advice for Windows Server owners is provided at the bottom of both articles.

---

> [heading] A never-ending arms race

Are there still protocols newly put into production that do not take reflection attacks into account? In my experience, it's all legacy. And don't more and more ISPs enforce BCP38 (http://bcp38.info)?

Not sure this headline of Ars' is true.

Am I the only disturbed by the amount of times the article used the word torrent for huge amount of traffic?

Trips me up as well, perhaps because I'm a non-native speaker. The mention of torrent in a networking-related context will immediately make me think of the bittorrent protocol.

In the days before we dealt with bufferbloat well using torrents was a great way to DOS one's self, especially if you had async internet service with a much slower upload or a router with terrible NAT that overflowed when a lot of IPs were in use. A torrent torrent would flood your equipment.

I bet they'd notice and shut them off pretty quickly if bandwidth wasn't nearly free.

I’m wondering about these religious organizations that have 10-20Gb/s pipes.

Some of the first high bandwidth customers that did video streaming I had to deal with were churches that were casting their sermons. While most people think of non profit organizations as poor that can be a mistake as many are very effective at collecting donations.

Sometimes non profits can get special deals. They can apply for tax exemptions (in the U.S.)

Why are people running Windows in the first place?

On a server it's just crazy. Microsoft has thoroughly proven that they have no clue how to make a mail server, remote desktop server or network services that can be safely connected to a public network.

On a PC it's also a privacy nightmare.

Active Directory, Exchange and to a lower level Citrix are the answer.

Very few businesses think they can do without those. IT history of these 30 last years have kind of segregated sysadmin/architects types. The one focused on internal IT / desktop services solutions are typically more knowledgeable in Microsoft technologies while unix/linux sysadmins usually focus on the non desktop/office related stuff. Despite decent solutions existing without Microsoft, very few people know enough about them to recommend them.

What is your AD replacement in Linux env?

samba?

Only talking about on-premises tools: Samba (if you need GPO supports for windows desktops), 389 Server, redhat directory server, openldap can work with Keycloak to provide saml2/openid authentication for individual applications.

Most emails clients support ldap for contacts with additional carddav server to sync personnal contacts accross devices, Sogo groupware works well for that , add a fairly decent webmail interface and has an outlook connector for easy configuration of microsoft desktops but if you need office 365 functionnalities, Nextcloud would be the way to go imho. It is available as a service too and is cheaper than Microsoft 365. I am not sure how an on premise Nextcloud would scale to huge companies (but enteprise offering say it scales to hundreds of millions of users) but for small businesses it is easy to set up.

I really only have experience administering windows AD desktop - mostly LDAP/GPO/DNS, some Share and thankfully no Print

I would really like some comparison from administration effort:

1. How easy it is to tie those things together? Like installing that Linux AD server components

2. How easy from deployment POV it is to join computers to network and start using a networked user account?

3. How easy it is to use print server connected to domain?

4. How easy it is for member servers within domain to start accepting SSO connections from those clients/servers?

5. How easy it is to have centralized configuration for linux host and target them? I mean using Linux tools as I suppose linux env won't have gpmc.msc. Let's say you would like all your client hosts, all software to talk TLS1.2+ only.

6. How easy would it be to establish internal certification authority with certificate auto renewal?

I'm aware of paid solutions, but just from marketing material, so no clue how it behaves in real life and how far it goes.

Please don't assume I'm starting flamewars. I just see a great value within Windows AD environment that is easy to set up but as we see from article, takes effort to configure and maintain secure and consistent environment.

My experience with Linux is limited in domain area, and I haven't been familiar with centralized configuration options - I just want to see if it takes 1x/2x/10x effort to do the same in Linux env.

OpenLDAP and Keycloack have docker images that can be used to deploy them pretty easily, at least if you're comfortable with container-based deployment. I knew nothing about both and able to deploy them in a few hours.

Connecting individual servers / applications to OpenLDAP and Keycloack varies widely though, not all apps has OIDC support but you can put them behind an authenticating proxy like oauth2-proxy.

Can't comment on how easy it would be to join desktops and print servers though as I never did it myself.

It is really complicated to answer.

There are things that are super easy. For example samba is a real dropin replacement for an AD, you can join windows computer to it very easily. Similarly, the bigger distros all have easy setup to enable joining a linux desktop to an Active Directory for auth.

I have used centralized configuration management tool such as puppet, cfengine, saltstack and ansible for years. I did the exercise of managing desktops with them and it was fairly easy. Hey I even deployed windows servers with foreman and puppet.

But for you it might be a nightmare to have to learn a new domain specific language.

> But for you it might be a nightmare to have to learn a new domain specific language.

Not particularly for me :) Just seeing value proposition for Windows AD - in Linux world, you have to glue together different and competing things for good or worse IDK, but windows just bring that integrated experience OOTB. But good that there exist options in Linux world. It is probably twice++ the effort to manage Lin along Win desktops, so I see why corps don't bother... because you still have to have those windows boxes. And that translates into MS AD for easy management and vast, better or worse, talent pool. I see Windows corps, where appropriate, deploy Linux Servers within infra, for example webservers as that translates into saving $.

Samba, yeah, it just really works and good that we have it. But it is only a part in corporate management toolbox. Even MS Active Directory is a part in corporate management toolbox as you still need something like InTune to manage endpoint installs, still some parts must be scripted, monitored, etc.

Ideally you move completely away from AD. I've used Pgina before for alternative auth methods when you have to deal with a dozebox but it has major issues that make it not worth it, so in reality you usually have a virtualized samba domain that pulls from FreeIPA (389 ldap underneath). (and if you have a windows admin on staff who can't write a GPO by hand you can virtualize the DC admin doze machine and ensure good rbac.)

Back in the days there was a saying "Nobody will lose their job for choosing IBM". I guess Microsoft has been the new IBM for a while. I'm just praying that their decline will be imminent and fast.

It's still sad that incompetent people get to choose rotten technology - often burning tax payers money (or even religious donations).

Microsoft has been the new IBM for a few decades already.

In fact, this is considered ancient history nowadays.

Reply on news.ycombinator.com