Hacker News Reader

Comment by ghshephard | Hacker News Reader

Easy DOS to block upstream, thankfully. Time seems like the sort of thing that a CoLo should provided to it's tenants. GPS time is likely good enough for 99% of sites, and, the final 1% who really require super reliable time, and need to be concerned about GPS spoofing/outages, can install a local atomic clock in their cabinet.

View on HN

How would you block it? Any packet with the source port of NTP should be dropped?

No. ISPs should not get into the business of default block rules. This gets into all sorts of problems. At what point does a block become unreasonable? What if a tenant needs multiple sites in sync -- they must run NTP between them (or PTP) in order to determine their drift from one another.

If you really want upstream blocking of traffic, the best thing to look towards is BGP Flowspec (RFC5575)

"How would you block it? "

The same way all Upstream providers blocks a DDOS - by characterizing the DDOS flow, scrubbing, and dropping any packet which matches that pattern.

The idea here, is that during a DDOS, where you might get a bit more NTP drops than normal, none of the local users should be impacted if they had a quality local time source.

"What if a tenant needs multiple sites in sync"

That's why you have a quality time source. By definition, they are "correct" to within any tolerance that all but CERN physics experiments care about, and they don't use NTP anyways.

See: http://static.googleusercontent.com/media/research.google.co... for an example of using synchronized time globally, without reliance on running NTP between sites.

In particular, check out section 3, "TrueTime"

It's trivial to add a UDP policer to ntp traffic to your ntp hosts. Also, how many people are exposing NTP externally on the same IP's they're running regular content (http/https) on? If you're allowing UDP/123 to your web servers, you need to fire yourself.

I guess what I'm trying to say is, you should already have this ACL'd on your own.

A UDP Policer doesn't help in a DDOS, because you need to block the traffic before it gets onto your circuit. By the time you start inspecting traffic, it's too late, you've already passed the traffic on your (presumably limited size) circuit.

The NTP DDOS doesn't require that the web server (or, indeed, any server) at the target be listening to UDP/123 in order to cripple them. You just need to use up their circuit capacity.

What I was trying (and failing) to suggest, is that when the Upstream starts scrubbing out the NTP DDOS, there is a reasonable chance, that during the event the downstream customers will start to see more NTP drops than they normally would. And that one way that customers who care about NTP greatly could mitigate that possibility, would be by having multiple stratum 0 sources locally - I.E. A GPS Antenna and Atomic Clock. I then took it a step further, and suggested that this would be a great service for the CoLo to provide, because they are downstream of any packet scrubbing, and would therefore be able to provide a reliable time source during a DDOS attack involving rogue NTP traffic, which might end up with some NTP packets being dropped by the upstream ISP who was scrubbing the DDOS off the circuit.

I was suggesting ways to prevent being part of the DDoS, not protect yourself from the DDoS. In other words, lock down your hosts so that they don't participate in the attack.

no major providers that I'm aware of support flowspec. it's also juniper only currently

Correct. nLayer used to but stopped because Juniper's implementation is buggy.

Less buggy, more "Could be broken by idiotic Cloudflare automation engineers."

Obviously such rules wouldn't automatically be in place; they'd have to explicitly be requested. And yes, it is a reasonable request to ask your upstream to add a few ACL lines.

Really? How many transit providers today will enter custom ACLs for customers? Almost none of them will do this. The only time they will is when it's disrupting the rest of their network and other customers are complaining.

Level3 was the last one I know that would put in an upstream ACL.

There is no other way to prevent a DDOS than to contact your upstream and say, "I'm getting 20 Gigabits of shit traffic that I'm not going to pay for. Make it go away."

Other than buying ports that are greater size than any potential DDOS, that is the only way you can defend against a flood of traffic.

There are services like Prolexic and Defense.net that you advertise your routes to, then they send you clean traffic.

Providers won't care if you asked for the traffic or not. You'll get billed. They won't put in an ACL. Not today. 5 years ago, maybe. Today, no chance. They'll want to sell you their own DDoS mitigation services (at hefty fees).

TWTC (and others) will sell you a "clean pipe" option which includes mitigation, and you only pay for what passes through mitigation. It's more expensive than regular transit, but it's an option if you want to keep things simple.

ACLs are not an option today.

For smaller webops shops, blocking ntp might not be possible.

For larger ones, having local strata0 timesources are an option. GPS gives you very precise, relativistic time for extremely low cost. (Bias: /me == Frmr Trimble employee.)

Reply on news.ycombinator.com