Route leak incident on October 2, 2014
blog.cloudflare.com
blog.cloudflare.com
Fragile in that any person who controls an Autonomous System can advertise routes and if your neighboring AS'es are not configured to filter out stuff it will just keep propagating. Imagine a government deciding to just reroute traffic to their data center maliciously. Obviously a good chunk of the internet will go WTF and blacklist the route but not before you get a deluge of data.
In the same way it is easy to route around traffic. Congestion in the midwest? No problem. Send the traffic down to Texas. All this is built on network admins trusting each other for the most part.
I'm sure someone posted a map on here showing that this happens about once a month.
BGPMon can tell you if this is happening to your AS
This is most of internet infrastructure. It was a more naive and happy time when the protocols that make up the internet were created.
Edit (clarification):
- BGP has no authentication so anyone can advertise 'anything' (still has to be a valid address).
- "it just comes" isn't entirely accurate as the changes propagate outwards through your peers (in p2p fashion). I'm not sure what happens technically when two networks are advertising themselves as serving a particular IP though (I have a fuzzy idea, but don't know how the edge-cases would be resolved).
From the perspective of a router trying to route packets, you basically make choices based on what your peer networks are advertising that they can (directly or indirectly -- e.g. route through) service.
[Disclaimer: This is all stuff that I "know" but I've never worked directly with BGP so I'm open to correction.]
Edit: I guess you could use IPSec. It's not perfect, but it's pretty cool that we could have end-to-end crypto at the IP layer. https://en.wikipedia.org/wiki/IPsec#Security_architecture
Of course most of this is mitigated by end-to-end crypto but given that we see all too frequently how fallible that can be, this topic remains of interest. I mean if crypto fails and leaks your private key (a la heartbleed) and it falls into the hands of an attacker who can hijack some BGP routes then that attacker is potentially in a very powerful position. We've seen BGP hijacking by spammers needing clean IPs in the past, so this isn't a totally implausible situation.
It is useful to remember that IP is connectionless protocol. The role if IP is just to shuffle packets around from node to node (where many of the nodes are not the source or final intended destination of the packets) in a fairly simplistic way.
> ... to some machine with an interface configured to that address
Technically, the other endpoint does not even need to have an interface configured with that address. You can quite easily configure a box to send replies for any packets that happen to end up to it.
> Oh dear.
There is a good reason why IPsec (etc) was invented.
If an ISP in Los Angeles transits five autonomous systems to reach an ISP in New York, but four to reach an ISP in London, and the London ISP suddenly starts advertising the New York ISP's IP addresses, oops, your packets are now headed to the UK.
Great care is taken to avoid bizarre physical routes for packets, but it still happens constantly. One DSL connection I used in Santa Clara had a propensity for routing packets through Seattle to get to Dallas for a while.
if an isp in india advertises all of amazons AWS ranges as /24 or even smaller the routes will go there even if the bigger ranges are closer.
It's common for one AS to advertise routes for locations not within their network. It's called transit.
http://www.dasblinkenlichten.com/bgp-route-filtering-access-... http://blog.ine.com/2008/01/08/using-extended-acls-for-bgp-f...
It's hard to be comfortable when this is true of systems as important as those which route the internet or PKI, for example, because it's impossible to know what might happen next. Perhaps it's erroneous to take the "structure" in "infrastructure" literally but in the context of the internet that word is becoming increasingly misnomered in my mind.
http://blog.cloudflare.com/why-google-went-offline-today-and...
> For the time being, we have quarantined the Medellín data center
> and disabled connectivity with Internexa.
I would have anticipated that Internexa's routes would have affected routing by third party networks (eg: Internexa claims it handles traffic for CloudFlare's IP range, and Level3 redirects all that traffic to them) which isn't something that CloudFlare can do much about other than notify Internexa and those third parties of the problem and hope they resolve it themselves. Having not worked directly with BGP, I'm sure I'm misunderstanding something and would appreciate any additional clarification.
edit: Incidentally, this is the kind of scenario which prevented me from using CloudFlare recently. I wanted to only CNAME our production web site to CF's systems which is something they only offer with the $200/month Business plan and not with the $20/month Pro plan. Otherwise, you have to delegate ALL of your DNS for the entire domain to CloudFlare. As one user in the comments says in response to how someone could have worked around the outage:
"That [bypassing CF] would be a good idea, except cloudflare.com
and control panel was inaccessible during this period too, so not
sure how this could have been done..."
"This downtime was the result of a BGP route leak by Internexa, an ISP in Latin America. Internexa accidentally directed large amounts of traffic destined for CloudFlare data centers around the world to a single data center in Medellín, Colombia. This was the result of Internexa announcing via BGP that their network, instead of ours, handled traffic for CloudFlare. This miscommunication caused a flood of traffic to quickly overwhelm the data center in Medellín. The incident lasted 49 minutes, from 15:08UTC to 15:57UTC."
The problem wasn't that cloudflair thought their IPs were somewhere else, the problem was that the rest of the internet[1] thought cloudflair lived in Medellin (somewhere else) and not with cloudflair.
[1] Complete hyperbole, but "huge swathes"[2] of the internet had the wrong idea.
[2] "The exact impact of the route leak to our customers’ visitors depended on the geography of the Internet. Traffic to CloudFlare’s customers sites dropped by 50% in North America and 12% in Europe. The impact on our network in Asia was isolated to China. Traffic from South America was also affected as data centers there had to cope with an influx of traffic normally handled elsewhere."
As you say, if Medellin was announcing to the world they are cloudflair, cloudflair ignoring that doesn't seem to help ;)
Each level of delegation must be cryptographically validated. Route announcements not signed using a certificate to which authority for that block has been delegated must be rejected.
You could sign that you trust your next-to-last hops to deliver traffic to you but transitive delivery isn't guaranteed and not every chain of valid hops is actually workable as a route capacity-wise. A completely signature-valid route with near 100% packet loss isn't much better than an undeliverable leak.
However, it pretty much requires everyone to do it.
For example, are we checking CRLs & would this slow propagation (and what if the server is down), what about countries unable to get CA generated certificates due to political embargos (on either their money or businesses are just restricted from doing business with them e.g. Iran, NK, etc), it would grant the US government even more control over the internet (as now they control many root DNS servers and root CAs) which they could use for military or political purposes, and so on.
I'm not saying that the concept doesn't have merit. BGP is quite evidently deeply flawed. However this solution has so many gotchas, question marks, and complexity to it you really need to dig deep down into the proposal before knowing if it is even a good plan.
Its very likely that I and especially the other non-hn folks could have missed this all together (checking cloudflare for an issue is likely my last check list item due to its amazing record).
However, we are also working to ensure that we don't need to use it.
iirc IPv4's "bloated" routing tables are getting harder to maintain.
- throw away "martians" (routes to known-reserved/bad parts of IP space)
- throw away stuff you know they shouldn't be announcing (your stuff basically)
other than that, I think you have to trust them when they say that they have a magic route to IP 16.0.0.1 which is shorter than any other you know about. (Since it might be true).