Dyn Analysis Summary of Friday October 21 Attack

It's probably temporary? Move to R53, then figure out what's needed to manage records on two providers (if only for internal processes). Top engineering teams aren't going to knee-jerk this, right? Or did Dyn show some unfixable incompetence?

To be fair Route 53 will split over other providers so you should be good in theory. But yeah if something more specifically targets Route 53 then that could be the same problem.

At least that's my understanding anyway.

Spotify clients can just go to the IP address?

DNS is a rather simple service that was always meant to be run internally.

The question you should ask is why did these companies used an external DNS in the first place?

It lower surface attack. With companies like Dyn, they are affected even when someone is targeting other sites, while with internal DNS servers that are only used by themselves they will be down only if someone is attacking them directly.

If someone is targeting them directly it doesn't matter much that DNS is up and running, their site is still down.

With only a modest simplification you can view security as ultimately just being a figure measured in dollars: "it costs an adversary $X to beat these countermeasures." Your goal in securing a system is not to push X to infinity, though that might be a reasonable goal (e.g. if you're a security researcher designing new crypto primitives). Instead your goal in engineering your company's security consists in evaluating the value $V of what you're securing, and then raising X until X > V. There are uncertainties in measuring X and V and in how attackers will view these tradeoffs and so forth, but it's nothing you can't account for by building in an engineering tolerance like X > 2V. The basic story remains.

Spotify simultaneously has large resources and offers a non-essential infrastructure service (music to listen to while you're doing something else). The V gained in DoSing them is very small. They got attacked anyway because they shared infrastructure with other companies, which pools the V together to create something much larger. Some attacker saw a case where V >> X and attacked it to great success until Dyn was able to bring up X again. During the interim, Spotify was down despite having V << X.

In short: Spotify probably can't do DNS better than Dyn, but they can do DNS better than the sort of people who have reason to attack them (presumably trolls, maybe some future hacktivist who doesn't like some business decisions they make, unscrupulous competitors). This attack was a wake-up call for them, "oh, if we're pooling with these other folks then we'll become targets of larger hacktivist attacks and state actors, who are not directly targeting us per se." Those attackers could presumably still take out Spotify's home-rolled DNS, but they have no real motivation to target Spotify in particular any more.

Perhaps their "internal" nameservers are just vanity nameservers hosted by someone else.

The approximate difficulty in running DNS server is the same as running a static HTML web server. Low difficulty.

If Spotify's networks are all down, what good would a functioning DNS do?

(And I know it's not that simple, but that's probably the basic reasoning behind it.)

(I work for CF)

Indeed, if you want the HTTP/HTTPS traffic to go through Cloudflare, the DNS must go through Cloudflare. There are generally two ways to set it up:

a) You move your DNS auth to Cloudflare and allow it to manage it.

b) You keep managing your domain yourself, and CNAME to Cloudflare. See: https://support.cloudflare.com/hc/en-us/articles/200168706-H...

What you should do depends on your setup and threat model. Do you fear DNS auth going down? Do you think your DNS will be a target? Do you use Cloudflare to hide your HTTP origin IP addresses?

For example, if you fear DNS auth going down, but you must use Cloudflare for HTTPS (say: for caching and SSL certs), then changing DNS off CF makes little sense. You already assume stability by expecting it to work HTTP layer.

If you think you can be a target of DNS attack, I'd say having multiple auth is unlikely to give you more mileage.

If you can afford disabling CF on HTTP layer, exposing your HTTP origin IP and want to have two different DNS auth providers, fine, you can do CNAME. But then you have three vendors to worry about, and problems with each can lead to trouble.

I think the question is about the initial infection.

As I understand it UPNP allows a device to negotiate a port map with a NAT box. But the developer of the IoT device would have to specifically want to map port 22 or 23. It's not something that would happen automatically.

So are you saying that these IoT device makers not only hard-coded root usernames and passwords into their devices, but deliberately set up UPnP mappings to those ports?

That looks malicious rather than negligent. Am I missing something?

I'm surprised there are UPNP implementations that even allow mapping to specific, low-numbered ports.

(Well, not surprised as much as disappointed.)

There are still many corporations and ISP's that haven't configured or support ipv6 yet.

OVH throws their DDoS mitigation system in for free when you host with them.

You would think that, yet Akamai dropped Krebs over half that.

In the case of a DNS amplification attack (like this one), a malicious host can cause much more traffic than it can itself supply.

"They mention 100k participating devices/IPs - that would mean an average of 12 Mbps upload"

I doesn't have to be. An attack can be highly asymmetric such amplification/reflection attack. I am not saying this is what happened in the Dyn case but rather addressing your average comment. Just an example having hosts in the botnet send queries to non-Dyn open resolvers on the internet and in those queries they spoof the source IPs of Dyn DNS servers. Now all of those open resolvers on internet start sending return traffic to Dyn. If in those spoof request they request EDNS0 those responses could be up to 4K. Compare that to the size of query packet and you have a lot leverage.

We already know Mirai has been able to reach over 1 Tbps, and we know Mirai was at least one of the cannons hitting Dyn. So 1.2 Tbps is definitely plausible. Mirai has decreased in size to a degree due to more public awareness, but it's still massive.

To cost the company significant amounts of money. Bandwidth in those volumes isn't cheap.

Exactly my point. There was no 'dns is slow but finally works now' moment, there was total dns blackout until attack stopped.

My company only just noticed our font renders terribly for one of our newer web apps - all the developers had Macs and didn't notice an egregious difference when rendered on lower quality displays. We changed it immediately. I feel empathy for this particular development oversight among small teams.

Yeah, a content font shouldn't be at 300. It's way too thin at that font size.