DDoS Attack Against Dyn Managed DNS
dynstatus.com
dynstatus.com
Ideally, then, the local resolvers of the nodes and/or the UIs of applications could detect the last-known-good flag on resolution and present a UI to users ("DNS authority for this domain is unresponsive; you are visiting a last-known-good IP provided by a resolution from 8 hours ago."). But that would be a nicety, and not strictly necessary.
Is there a spectacular downside to doing so? Since the last-known-good resolution would only be used if a TTL-specified refresh failed, I don't see much downside.
The scenario is that my local network's caching DNS resolver retains resolutions beyond the authority-provided TTL in the event that a TTL-specified refresh at expiration fails. Therefore, my web browser may—in the very rare situation where this arises—make an HTTP request to an IP address of a server that has been intentionally moved by a service provider (let's assume they did so expecting their authoritative TTL to have expired). Since this scenario only arises because my caching resolver wasn't able to reach the authority, I'm not seeing a downside.
But if I understand your reply correctly, you are saying that the web server I've contacted may, in turn, be using a DNS resolver that is similarly configured to provide last-known-good resolution when its upstream provider/authority cannot provide resolution. This would potentially result in that web server making an HTTPS API request to a wrong IP, again only in the rare case where we have defaulted to a last-known-good resolution. I'm not really seeing the problem here except that the HTTPS request might fail (if the expected service was moved and no longer exists at the last-known-good IP), but how is that worse than the DNS resolution having failed? In both cases, the back-end service request fails.
I mean, hopefully it is over HTTPS so they can't do anything with it... but if it isn't then it can definitely happen. Our servers get random web traffic all of the time.
DV certs only rely on you being able to reply to an HTTP request, so if any CA was using such a caching DNS server, you could probably get a valid cert from them.
If you have a service `log.io` with it's own DNS servers ( running named or djbdns ). And one day you decide to shut them down and rename the service to `loggy.io`.
What will happen is that any DNS trying to query the `log.io` DNS will reach unreachable server, which will lead to serving the last-known IP from the proposed DNS Cache on your machine.
If you don't use forever-failback-cache after the TTL expired you will just reach unreachable server and return back no IP address.
> If you have a service `log.io` with it's own DNS servers ( running named or djbdns ). And one day you decide to shut them down and rename the service to `loggy.io`. > What will happen is that any DNS trying to query the `log.io` DNS will reach unreachable server, which will lead to serving the last-known IP from the proposed DNS Cache on your machine.
To reiterate the scenario you've put forth as I understand it: I'm a service operator and I've just renamed my company and procured a new domain. I've retired the old domain and expect to fulfill no more traffic sent to that domain. When a customer of my service attempts to resolve my old domain, their caching DNS resolver may return an IP address even though I have since shut down the authoritative DNS servers for the retired domain. They will make an HTTPS request to my servers (or potentially someone else's, if I also gave up my IP addresses), and fail the request because the certificate will be a mismatch.
The customer's application will see a failed request either by (a) DNS failing to resolve or (b) HTTPS failing to negotiate. Either way, my customer needs to fix their integration to point to my new domain.
To be clear, it is up to my customer to decide whether they want their caching DNS resolver to provide a last-known-good resolution in the event that authoritative servers are unreachable. If they prefer failure type (a) over (b), they would configure their DNS resolver to not provide last-known-good resolutions.
Anyone sane will keep the domain name and ns infrastructure and serve a 301 HTTP redirect.
All anyone is proposing here is to override the TTL to something longer (like 48h) if the nameserver is unreachable.
Of course the perfect solution would be to have the recursive nameserver fetch the correct record from a blockchain.
you could write a cron script that generates a date-stamped hosts file based on a list of your top-used domain names, and simply use that on your machine(s) if your dns ever goes down. that's basically a very simple local dns cache.
if you feel like living dangerously, have it update /etc/hosts directly.
Probably because people used to use long TTLs (1 hour, 4 hours, whatever) and now the default behavior in services like Amazon Route 53 is to use 5 minutes.
http://www.infoworld.com/article/3133104/mobile-technology/w...
I wouldn't call DNS based IP mapping "horrendous" simply because it doesn't work as well for mobile,I understand you have your own pitch but lets go easy on the hyperbole :)
The fact is that it is still very effective. The major CDNs are quite aware of the mobile shortcoming of DNS based mapping and I am pretty sure it is something they are working to address.
At the end of the day location is just one component involved in accelerating content, there are plenty of other features that various CDNs use to deliver optimal performance.
Regarding the short TTLs, I get your argument, it is indeed like a user's browser is constantly chasing a moving origin. The alternatively however is a non-optimized web, which would be orders of magnitude worse. Remember the benefits of CDNs doesn't just accrue to end users but also to content providers, most origin servers can't handle even the slightest up tick in traffic.
OK I'll take back the word "horrendous" but it's a hack alright.
> The fact is that it is still very effective. The major CDNs are quite aware of the mobile shortcoming of DNS based mapping and I am pretty sure it is something they are working to address.
No not really. They're certainly trying to patch DNS to pass through enriched information in DNS requests through recursive calls... but it's such a long shot to work consistently across tens of thousands of networks around the world, and requires coordination from so many different entities, that it's clearly a desperation move more than than a serious effort. Regardless, there's no real solution in sight for the web platform.
For mobile (native) apps though, the right way to discover nearby servers is to directly build in that functionality using mobile specific techniques. There's no reason to keep limiting mobile apps to old, restrictive web technologies considering that apps have taken over as majority of traffic around the world. That's the root idea behind a lot of what we're doing at PacketZoom. Not just in service discovery, but also in more intelligent transport for mobile with built-in knowledge of carriers and network technologies etc, automatic load-balancing/failover of servers and many other things. Here's my older article on the topic
http://www.infoworld.com/article/3016733/application-develop...
Because you would keep old DNS records around forever if a server goes away for good. So you need to have a timeout for that anyways.
1) Memory and disk are cheap. My caching DNS resolver can handle some stale records.
2) I suggested above that this behavior would continue until an administrator-specified and potentially quite generous maximum TTL expires. That is, I could configure my caching DNS resolver to fully purge expired records after, for example, 2 weeks.
The problem is not that it would require storage but that stale records can be outright wrong. That timeout would require configuration and DNS does not provide that.
So sure, a new timeout could be introduced but that currently does not exist in DNS.
Again, the scenario is that the authoritative/upstream resolver cannot be reached in order to refresh after the authority-provided TTL expires. Are you saying that in the case of a service having been intentionally removed from the Internet (the domain is deactivated; the service is simply no more), my caching resolver will continue to resolve the domain for a time? Yes, it would. What's the downside though?
> That timeout would require configuration and DNS does not provide that.
Yes. This would be a configurable option in my caching DNS resolver, in the same vein as specifying the forwarders, roots, and so on. But to be clear, this would not be a change to the DNS protocol, merely a configuration change to control the cache expiration behavior of my resolver. I'm not wanting to sound flippant, I'm not sure I understand the point you're trying to make here.
But the tradeoff here is a wrong record vs a complete failure to lookup the record. I would rather have the wrong one.
The domain expires in January. I hope it's not set to auto-renew. :-)
That's the kind of world we used to live in when TTLs were often treated as vague suggestions.
I believe the scenario you are describing is a rogue ISP ignoring that authoritative TTL wholesale, caching resolutions according to its own preferences regardless of whether the authority is able to provide a response after the authoritative TTL expires.
Among other problems, this enables attacks. Leak a route, DDoS a DNS provider, and watch as traffic everywhere goes to an attack server because servers everywhere "protect" people by serving known-stale data rather than failing safe.
Be very, very careful when trying to be "safer". It can unintentionally lead somewhere very different.
Maybe you can enlighten me on key differences I've overlooked? How do you define "failing to reply"? Do you ever stop serving records for being stale, or do you store them indefinitely?
The older I get in tech the more I realize we just go in circles re-implementing every bad idea over again for the same exact reasons each "generation". Ah well.
TTL is TTL for a reason. It's simple. The publisher is in control, they set their TTL for 60 seconds so obviously they have robust DNS infrastructure they are confident in. They are also signaling with such low TTLs that they require them technically in order to do things like load balance or HA or need them for a DR plan.
Now I get a timeout. Or a negative response. What is the appropriate thing to do? Serve the last record I had? Are you sure? Maybe by doing so I'm actually redirecting traffic they are trying to drain and have now increased traffic at a specific point that is actually contributing to the problem vs. helping. How many queries do I get to serve out of my "best guess" cache before I ask again? How many minutes? Obviously a busy resolver (millions of qps at many ISPs) can't be checking every request so where do you draw the line?
It's just arrogant I suppose. The publisher of that DNS record could set a 30 day TTL if they wanted to, and completely avoid this. But they didn't, and they usually have a reason for that which should be respected. We have standards for a reason.
Here's the attack:
- Compromise IP (maybe facebook.com)
- DDoS nameservers
- facebook removes IP from rotation
- Users still connect to bad actor even though TTL expired
"We have standards for a reason" is absolutely correct, and we can't start ignoring the standards because someone can't imagine why we need them _at this moment_
> Here's the attack:
> - Compromise IP (maybe facebook.com)
- Attacker generates or acquires counterfeit facebook.com certificate.
> - DDoS nameservers
> - facebook removes IP from rotation
> - Users still connect to bad actor even though TTL expired
I understand what you are saying, but this attack scenario is extraordinarily difficult as a means to attack users who have opted to configure their local DNS resolver to retain a last-known-good IP resolution. It involves commandeering an IP and counterfeiting Facebook's SSL/TLS certificate. As I have said elsewhere in this thread, all sites are currently vulnerable to such an attack today for the duration of their TTL window. So if this is a plausible attack vector, we could plausibly see it used now.
This is why some people are concerned about technical decisions that make this vector more dangerous. Systems that attack by, say, injecting DNS responses already exist and are deployed in real life. The NSA has one - Quantum. Why make the cache poisoning worse?
If my adversary can steal an IP from Facebook, create a valid certificate for facebook.com, and provide bogus DNS resolution for facebook.com, I feel it's game over for me. My home network is forfeit to such an adversary.
But I get your point. It's about layering on mitigating factors. The lower the TTL, the lower the exposure. Still, my current calculus is that the risk of being attacked by such an adversary is fairly low (well, I sure hope so), and I would personally like to configure my local caching resolver to hold onto last-known-good resolutions for a while.
All that said, I have to hand it to you and others like you, those whom keep the needle balanced between security and convenience.
Keeping the balance between security and convenience is difficult on the best of days. Today is not one of them. :/
So you enabled an attack vector that has to be nullified by a deeper layer of defense? And in some cases possibly impacted by a user having to do the right then when presented with a security warning.
Why would you willingly do that?
Also I do find your assumption of ubiquitous TLS rather alarming - facebook is a poor example here, there are far softer and more valuable targets for such an attack vector to succeed.
Edit: Also to keep my replies down...
> I would personally like to configure my local caching resolver to hold onto last-known-good resolutions for a while.
You can! All these tools are open source, and there are a number of simple stub resolvers that run on linux (I'd imagine OSX as well) which you can configure to ignore TTL. They may not be as configurable as you like, but again they are open source and I'm sure would welcome a pull request :)
Hang on a second. I feel that you're piling on other resolver changes in order to make a point. I'm not suggesting that the tolerance for DNS response times be reduced. Nor am I suggesting a scenario where the authority gets one shot after their TTL, after which they're considered dead forever. I would expect my caching DNS resolver to periodically re-attempt to resolve with the authority once we've entered the period after the authority's TTL.
> Leak a route, DDoS a DNS provider, and watch as traffic everywhere goes to an attack server because servers everywhere "protect" people by serving known-stale data rather than failing safe.
I think you're suggesting that someone could commandeer an IP and then prevent the rightful owner to correct their DNS to point to a temporary new IP.
Isn't the real problem in this scenario the ability to commandeer an IP? The malicious actor would also need to be able to provide a valid certificate at the commandeered IP. And at that point, I feel we've got a problem way beyond DNS resolution caching. Besides, if what you have proposed is possible, isn't it also possible against any current domain for the duration of their authoritative TTL? That is, a domain that specifies an 8-hour TTL is vulnerable to exactly this kind of scenario for up to an 8-hour window. Has this IP commandeering and certificate counterfeiting happened before?
Yes. The point I am making is the additional failure modes that need to be considered and the pain they can cause. Historically have caused.
At no point did I ever think you were suggesting that one failure to respond renders a server dead to your resolver forever. Instead, I expect that your resolver will see a failure to respond from a resolver a high percentage of the time, leading to frequent serving of stale data.
> Isn't the real problem in this scenario the ability to commandeer an IP?
You're absolutely right! The real problem here is the ability to commandeer an IP.
However, that the real problem is in another castle does not excuse technical design decisions that compound the real problem and increase the damage potential.
If this were true, the current failure mode would have end users receiving NX DOMAIN a "high percentage of the time," which obviously is not happening.
{edit: To be clear, I'm reading the quote as you stating that "failure to resolve" currently happens a high percentage of the time, and therefore this new logic would result in extended TTLs more often than the original post would assume they would happen}
> However, that the real problem is in another castle does not excuse technical design decisions that compound the real problem and increase the damage potential.
It's fair to point out that this change, combined with other known issues could create a "perfect storm," but as was pointed out this exploit is already possible within the current authoritative TTL window. Exploiting the additional caching rules would just be a method of extending that TTL window.
On the other hand, where do you draw the line here? If you had to make sure that no exploits were possible most of the systems that exist today would never have gotten off the ground. It seems a bit like complaining that the locks to the White House can be exploited (picked), while missing the fact that they are only supposed to slow someone down before the "men with guns" can react.
I don't need to make sure no exploits are possible. However, it at all possible, I'd like to help ensure that things aren't accidentally made more dangerous. It's one thing to consider and make a tradeoff. It's quite another to be ignorant of what the price is.
1. Let's somehow get a record that points at a host controlled by us into many resolvers (by compromising a host or by actually inserting a record).
2. Let's prolong the time this record is visible to many people by denying access to authoritative name servers of a domain.
(1) is unrelated to caching-past-end-of-ttl, so you need to be able to do (1) already. (2) just prolongs the time (1) is effective and required you to be able to deny access to the correct DNS server. Is it really that much easier to deny access to a DNS server than it is to redirect traffic to that DNS server and supply bogus reponses?
Ignoring TTLs in favor of your own policy means poisoned DNS caches can persist much longer and be much more dangerous.
In that world, one can still do that. One can also poison the entry once and then deny access to the real server. You seem to be arguing that this is easier than continuous poisoning. Do I understand you correctly?
I am in no way arguing about ease of any given attack over any other. I am arguing that a proposed change results in an increased level of danger from known attacks.
I'm arguing that the proposed change at hand, keeping DNS records past their TTLs, makes DNS poisoning attacks more dangerous because access to origin servers can be denied. Right now TTLs are a real defense against DNS cache poisoning, and the idea at hand removes that in the name of user-friendliness.
You are arguing that a kind of attacks is made more dangerous, because in the world with that change an attacker can not only (a) keep performing attack X, but can also (b) perform attack X and then keep performing Y. If Y is in no way simpler for the attacker why would an attacker choose (b)? S/he can get the same result using (a) in that world or in our world.
Am I misreading you or missing some other important property of these two attack variants?
X cannot always be done reliably - it usually relies on timing. Y, as we've seen, can be done with some degree of reliability. Combining them, in the wished-for world, creates a more reliable exploit environment because the spoofed records will not expire. The result is more attacks that persist longer and are more likely to reach their targets.
Such a world is certain to not be better than this one and likely to be worse.
Also, "commandeering" an IP of a small hosting might be easier than you think. It depends entirely on how they recycle addresses.
This proposal would allow an attacker to prolong the effects of cache poisoning by running a simultaneous DDoS against un-poisoned upstream DNS servers.
You likely underestimate the sheer number of DNS records you look up just by surfing the web, and how useful that information would be to 99.99% of users.
Basically the tools exist for you to do this yourself if you are so inclined, but they may not be that user friendly since they aren't generally useful to most.
It's called SmartCache.
https://www.opendns.com/enterprise-security/technology/globa...
(Reluctantly in that Google already has enough of my data, thanks, through gmail, search, maps, docs and other services, not because it doesn't work well.)
Still, I prefer it to isps snooping.
$ dig -tA twitter.com @208.67.222.222
; <<>> DiG 9.8.3-P1 <<>> -tA twitter.com @208.67.222.222
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63973
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;twitter.com. IN A
;; ANSWER SECTION:
twitter.com. 0 IN A 199.59.148.82
twitter.com. 0 IN A 199.59.149.198
twitter.com. 0 IN A 199.59.148.10
twitter.com. 0 IN A 199.59.150.7
;; Query time: 14 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Oct 21 11:53:40 2016
;; MSG SIZE rcvd: 93
$ dig -tA twitter.com @8.8.8.8
; <<>> DiG 9.8.3-P1 <<>> -tA twitter.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47295
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;twitter.com. IN A
;; Query time: 13 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 21 11:53:47 2016
;; MSG SIZE rcvd: 29Say, generating corporate communications seems like a promising direction for neural networks. A Markov chain comes close...
Ironically, those unrealistic expectations are probably a significant factor in the growth of data mining and resell; how else is a free-to-use website that doesn't have any ads (or whose users mostly block ads) going to get paid? You may say "not my problem", but it affects you when you leave the company no option but to resell data on the behaviors they observe from you.
I thought it was great. 10,000 companies pay for my service today. 65 million people use my infrastructure today. Cisco bought the company for more than $650m. It continues to innovate on the decades old DNS in secure and useful ways.
So let me know what part is terrible.
NXDOMAIN. Kind of a thing, and important to protocols other than HTTP.
If what they do is useful to you but have a feature or bug or something else you don't like then you absolutely should forgive them if they then fix that feature or bug to work in a way you like. They may as well never bother fixing things if they can never be forgiven after repenting their internet sins.
If you've since found something that does do what you want then fair play, fill your boots. Otherwise you're being petty for the sake of being petty.
It's called HOSTS and djb's cdb constant database.
And one does not need to use a recursive cache to get the IP addresses. Fetching them non-recursively and dumping them to a HOSTS and a cdb file can sometimes be faster; I have a script that does that. Fetching them from scans.io can be even faster.
cd||exit
[ -c null ]||mknod null c 2 2
case $# in
0)
{
sed '
/#/d;
/^[0-9]/!d;
' /etc/hosts \
|{
while read a b c d;
do
echo +${#b},${#a}:$b-\>$a;
done;
}
echo;
} \
|exec awk '!($0 in a){a[$0];print}' \
|exec cdbmake $0.cdb $0.t||exit
exec cdbdump < $0.cdb
;;1)
test ${#0} = 2 ||
exec cdbget $1 < $0.cdb >null;
exec cdbget $1 < $0.cdb;
esac
usage: $0
usage: $0 domainname
With all due respect to the enormous reliance on it that has built up over the past decades, DNS is not the internet. It is just a service heavily used for things like email and web. This does not mean, in an emergency, email and web cannot work without DNS. They once did and they still can.
The internet runs just fine without DNS. Some software may refuse to honour HOSTS and rely on solely on DNS. But that is a vulnerability of the software, not the internet. (And in such cases, e.g., qmail, I just serve my own zone via tinydns, which again is just a mirror of HOSTS.)
For one, how are you doing to deal with stale records?
If you have any feedback, I'd love to hear it.
Serving data you cannot verify is a dangerous failure state.
But I have to wonder about situations like this where DNS has been taken down, what the better good is. If the records can be proven to have been cached as authentic data at some point within some period of time. In this case hours, is it for the better good that stale authentic records are acceptable to serve back? In this case a stale period of some number of hours would have been good.
I'm not so sure which is better in this case.
As another user put it, we have these standards for good reason.
EDIT: It would be fine as long as your site only served HTTPS content and HSTS was enabled for your domain, preventing any sort of MITM attack.
They'd need to specifically gain access to the last known good IP address, which might be different depending on which DNS resolver you talk to (geodistribution, when the record was last updated, etc). I wouldn't really consider that a realistic attack vector.
Don't want to say much more due to it being my job, and I don't want to give away too much.
It provided value by answering my question concerning serious downsides to providing optional post-TTL last-known-good caching within a DNS resolver. The answer is implicit in that a major DNS resolver provides exactly this functionality.
A little more information, considering it is public. (I had to double check if it was)
Also, that cache would need to only kick in when the server was unreachable or produced SERVFAIL, not when it returned a negative result. Negative results returned by the authoritative server are correct, and should not result in the recursive resolver returning anything other than a negative result.
Precisely. I am not suggesting any change to how a caching resolver comprehends valid responses from the authoritative server for a domain. For example, if the authoritative server says, "No such domain," then the domain is understood to be gone. At that point, the domain being gone is in fact the last-known-good resolution.
https://www.schneier.com/blog/archives/2016/09/someone_is_le...
Edit: And to be clear: I don't mean to imply there's any connection :)
Why not the USA?
Note that this wouldn't rule out the USA as such. First, it could be a longshot preparedness thing, with no expectation that it would ever be used. Second, they could be red-teaming the thing (looking for weaknesses so that they can arrange for them to be shored up).
In either of these scenarios, it's no less likely that the USA would be doing it than anyone else. If you assume that whoever is doing this is planning to use their knowledge, however, the economic argument makes the USA less likely to be involved.
I'd be more willing to put my money on someone attacking an entity downstream who is normally immune to DDOS attacks of this size.
For example, if a particular part of the government got wind of a data dump about to be released by another nation-state or independent actor (for example, a leak of some kind) - I think some parts of the USA government that possesses the ability to do so wouldn't hesitate to take down dns to the entire internet to avoid another similar data leak to the Snowden dump.
Be really wary of attributing intent: you do not know who will benefit the most from taking down certain services. To claim that the US benefits from the internet so much that it wouldn't do certain actions to protect itself from certain types of harm is shortsighted.
Even my example could be really wrong, but the idea is that nobody really can say - "oh the internet is too important to xyz, they'll never do anything!"
I don't understand how this would change anything unless you're assuming they would take down the Internet permanently
Remember, a data leak is not just a technical issue. They can resolve it in any number of ways - get a small team incursion into another state's territory for extraction, etc. All the outage needs to do is to hold open that window for enough time for all the different parts of the entire threat response chain to do each part's job.
A lot of technical people think tech is the end, but no - if you get a small team to go knock on the person's door, and get your internet response team to shut down dns, or to get someone on site at the telco to perform certain actions at the router/switch level, etc all portions working together is a powerful way to resolve or to accomplish certain goals.
Think bigger, especially with state actors - the resources are there, and this line of thought is probably really basic stuff that people came up with in the 1960's or 70's (even when the arpanet was being created, there was probably already a team tasked with taking such actions - it only make sense to have 2 teams working on such goals in tandem - one to create the network, the other to take it down)
News cycles happen fairly rapidly, so if you could take down a number of sites that might be friendly to the dissemination of potentially damaging information just long enough such that it's forgotten about, or the attack is so large the media talks about the attack instead, then you might be able to successfully avoid widespread public knowledge of such information. Though, this would be best aided with collusion or cooperation (intentional or otherwise) from the media. Toss in a few unrelated services as a bonus for collateral damage, and you might be able to avoid scrutiny or, at the very least, shift the blame to an unrelated state actor. It won't prevent the release of information, but that's not the point--you want to prevent the dissemination and analysis of that information by the public at large.
This is all hypothetical, of course, and not likely to work. It also comes with the associated risk that if you were discovered or implicated, public outrage might be even worse than if you allowed the release of the information you hoped to distract from in the first place! As such, I can't imagine anyone would be stupid enough to try.
I'll take my tinfoil hat off now.
I still give it less than a 5% probability, though.
Honestly, that's fairly thin. WL uses torrents and other means of disseminating data that don't rely on central control structures. Plus, presumably, WL has the ability to quickly shift data into secure hands who are willing to release it when things quiet down.
So, sure, the USA could go send someone to sieze the hard drives of someone who has confidential information. But, I have to imagine one of the first steps when getting that kind of information is to disseminate it to others (at least some of whom are unknown to the states). If they were hit, these people would very quickly take that as a signal to indiscriminately release all the information.
Not necessarily so strange. See https://en.wikipedia.org/wiki/Bootleggers_and_Baptists for instance.
As for the length of the "test," they might want to see how the US would react to such attacks in the future, and shake out anything critical. "Oh, these two agencies can't talk to each other. Good to know."
I hate the way modern times makes me look.
To pin it on someone else?
"17 Intelligence agencies told me Russia hacked our DNC thing" (Clinton).
So maybe it is now "Oh look they took down the whole internet as well".
That's... kind of conspiratorial thinking? Would you cut off your own hand so you could blame it on someone else?
Where did I say it was my immediate explanation and this is _likely_ what is happening?
> kind of conspiratorial thinking?
You mean like lizard aliens infiltrating our planet? -No. But in the realm of "shooting down of passenger and military planes, sinking a U.S. ship in the vicinity of Cuba, burning crops, sinking a boat filled with Cuban refugees, attacks by alleged Cuban infiltrators inside the United States, and harassment of U.S. aircraft and shipping and the destruction of aerial drones by aircraft disguised as Cuban MiGs", yes.
It was mostly a reply to "US would have absolutely no reason for doing this" and the reply is there cold be a plausible reason.
Then I felt horrible.
Some NPR story about US Cybercommand responding to Russian cyber attacks, 'at place and time of our choosing.'
'Some you might hear about. Some you might not.'
FFWD to a couple days ago, NPR story about a botched European and Russian lander.
Today, US Eastern Seaboard is seeing connectivity disruption due to DDoS attacks.
*
Unwinding the stack, the latest news is these DDoS attacks are not likely state sponsored.
Russia pulling off a coordinated attack that soon after and in response to my theorized US retaliation seems unlikely.
US attacking a joint partnership between Russia and Europe civilian space program seems unlikely.
https://www.techdirt.com/articles/20160912/16553435504/fbi-d...
Of course, he said nothing about internal rigging:
Though if they targeted electric grid, water, and public transport, starting early in the day and choosing the regions by their populations political leaning, it could easily have an effect on the result itself.
Control the message, and through that the actual votes cast.
Of course, I have no information on the security model of the pre-election preparations and post-election tabulation, but luckily results for each polling place are also posted for the public to inspect - media outlets and campaigns can verify the tabulation themselves with a slight delay.
host -t ns twitter.com: ns3.p34.dynect.net, ns4.p34.dynect.net, ns1.p34.dynect.net, ns2.p34.dynect.net.
host -t ns amazon.com: ns3.p31.dynect.net, ns4.p31.dynect.net, ns2.p31.dynect.net, pdns6.ultradns.co.uk, pdns1.ultradns.net, ns1.p31.dynect.net.
"How do all these major players have singly-homed DNS"?
http://www.cnbc.com/2016/10/21/major-websites-across-east-co...
Is this par for course for all large DDOS attacks or did something tip them off?
I would assume that when a large number of big enterprise-y things go down, HSI takes notice. When other providers get attacks that are 20x larger (gbit/sec), but have much less widespread impact and impact on less enterprise-y things, they don't care so much.
As @scrollaway mentioned, 6 weeks ago, Bruce Schneier posted that several companies told him that they're detecting attempts to probe their networks and find ways to bring it down https://www.schneier.com/blog/archives/2016/09/someone_is_le...
Now let's look at the progress of events:
- Hillary Clinton's personal email server was hacked a while ago.
- A lone hacker published a document obtained by hacking the DNC servers. The document includes opposition research on Donald Trump and how Hillary can attack him in the election.
- Wikileaks published emails obtained by hacking the DNC
- US intelligence agencies confirmed that Russia was behind the DNC hack
- It was reported that the CIA is starting a cyber attack against Russian targets. http://www.nbcnews.com/news/us-news/cia-prepping-possible-cy...
- This is happening while the war in Syria and Iraq is growing. The Russians are there to "fight ISIS" but they have deployed an air defense system even though ISIS doesn't have any air force.
- Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.
https://www.theguardian.com/world/2016/oct/20/russian-fleet-...
https://www.theguardian.com/world/2016/oct/19/convoy-of-russ...
- Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.
http://www.businessinsider.com/r-finland-sees-propaganda-att...
- US ships were attacked near Yemen after they're bombed some targets the belong to the rebels. https://www.theguardian.com/us-news/2016/oct/13/us-enters-ye...
- US election is in 3 weeks and Donald Trump is openly in love with Putin. Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.
Say Hello to World War III, everybody!
Thank you for these links. I'm trying not to get wrapped up in conspiracies but am increasingly worried by the mounting conflict. I'd love to hear a calm, reasoned response from someone more knowledgable than me on these topics.
My personal opinion is that it is mostly political and I think (hope) that what is happening in Syria won't escalate to direct conflict between the US and Russia.
I stumbled across this little blog article the other day and it helped relieve some of my anxieties.
The players are the 0.001% who control these states and the rest (we) are the captive (and propagandized) audience. They are being super kind as to at least make it entertaining for us.
If they were really gearing up for war why would they move their only carrier away from the mother land. Your article even says it is more of a "show of force" than start of war.
So how did you jump to WW3?
It seems incredibly unlikely that a global war would start over Syria when we've had 60 years of proxy conflict instead. Russia or NATO have absolutely nothing to gain from an open military conflict.
He states that he's never met Putin nor has any holdings in Russia. He has stated that he is open to positive relationships with the Russian government.
> Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.
I believe he stated that he wants NATO to "pay their fare share" in the costs of maintaining the organization.
I'm not a Trump supporter but we shouldn't believe everything we read.
“I got to know him very well because we were both on ‘60 Minutes,’ we were stablemates, and we did very well that night.”
The Finns actually have still quite good relationship with Russia (better than other neighbors) and nobody's actually questioning Finland's independence. Baltic countries is a different story.
Source: A Finn here.
> Finland is becoming increasingly worried about what it sees as Russian propaganda against it, including Russian questioning about the legality of its 1917 independence
Is this sabre rattling or the prelude to a global conflict? Surely at worst it will (continue to) be a proxy war between NATO and Russia in Syria and nothing more? What motive is there for Russia or NATO to engage in open warfare? I'm not sure that a slow and prolonged lead up to an open war would even be effective in this situation.
Perhaps it should be "Say hello to Cold War v2.2017"?
I read they were passing in international waters. Is that not the case? It's clearly a show of force, but no need for the hyperbole if it is not true.
Finland isn't worried, they have had stable relations for half a century as both sides agreed to not mess with each other. They have even refused to join NATO because it is actually safer for Finland and vice versa.
Cowboys with missiles stationed on Russia's border making hyperbole statements (like you do) - now that would be a real threat. (the same was also true the other way around with the Soviets stationing missiles in Cuba)
If this is another practice run, then I'm still not impressed. Taking down one provider is not that hard. Good luck finding the resources to do this DDoS to ALL large DNS providers out there.
Maybe it's not really fair to link to that post every time a DDoS with more than average payload happens. Especially since the post doesn't mention any specifics, because well, "protect my sources". It's like the "buy gold now" guy starring in the 2 AM infomercial predicting an economic recession within the next 5 years, without adding what the exact cause is going to be. He is probably going to be right, but that doesn't make him a visionary.
Yeah it's not the whole Internet, but how do you define "taking down the Internet" anyway. Is it every connected computer or just a huge amount of interconnected big websites? Because the latter is happening right now.
If I had to guess, though, I don't think it's China. I think it's more likely related to the DDoS attacks against Brian Krebs than the probing attacks against the Internet infrastructure, despite how prescient that essay seems right now. And, no, I don't think China is going to launch a preemptive attack on the Internet."
[1] https://www.schneier.com/blog/archives/2016/10/ddos_attacks_...
In addition you can reach out to our customer support team at support@pagerduty.com or +1 (844) 700-3889.
Tim Armandpour, SVP of Product Development, PagerDuty
Specifically, they shouldn't have all of their DNS hosted with one company. That is a major design flaw for a disaster-handling tool.
I ask because my secondary question, as a network noob, is was anybody prepared / preparing for a DDOS on a DNS like this? Were people talking about this before? I live in Mountain View so I've been thinking today about the steps I and my company could take in case something horrifying happens - I remember reading on reddit years ago about local internets, wifi nets, etc, and would love to start building some fail safes with this in mind.
Two pronged comment, sorry.
Re question #1, check out PagerDuty's reliability page here: https://www.pagerduty.com/features/always-on-reliability/
Namely "Uninterrupted Service at Scale - Our service is distributed across multiple data centers and hosting providers, so that if one goes down, we stay available."
It seems fair to expect them to have a backup dns too, but I am not an expert.
Yes.
I have, personally, been under attack with as-large or larger than todays attacks at my DNS infrastructure and survived.
Knocking half of the web off the grid because their DNS provider is under attack? It happened recently to DNSimple.
https://blog.dnsimple.com/2014/12/incident-report-ddos/
The irony is that I noticed it when dotnetrocks.com went offline, at that time dotnetrocks was sponsored by dnsimple...
Calling it a "challenge" implies that there is some difficult, but possible, action that the customer could take to resolve the issue. Since that is not the case, this means either you don't understand what's going on, or you're subtly mocking your customers inadvertently.
Try less to make things sound nice and MBAish, and try more to just communicate honestly and directly using simple language.
Even a single quad-core server with 4GB RAM running TinyDNS could serve 10K queries per second, based on extrapolation and assumed improvements since this 2001 test, which showed nearly 4K/second performance on 700Mhz PIII CPUs: https://lists.isc.org/pipermail/bind-users/2001-June/029457....
EDIT to add: and lengthening TTLs temporarily would mean that those 10K queries would quickly lessen the outage, since each query might last for 12 hours; and large ISPs like Comcast would cache the queries for all their customers, so a single successful query delivered to Comcast would have (some amount) of multiplier effect.
See MaxCDN for example who uses a mix of dns providers (AWS Route53 and NS1):
ns-5.awsdns-00.com. ['205.251.192.5'] [TTL=172800]
ns-926.awsdns-51.net. ['205.251.195.158'] [TTL=172800]
ns-1762.awsdns-28.co.uk. ['205.251.198.226'] (NO GLUE) [TTL=172800]
ns-1295.awsdns-33.org. ['205.251.197.15'] (NO GLUE) [TTL=172800]
dns1.p03.nsone.net. ['198.51.44.3'] [TTL=172800]
dns2.p03.nsone.net. ['198.51.45.3'] [TTL=172800]
dns3.p03.nsone.net. ['198.51.44.67'] [TTL=172800]
dns4.p03.nsone.net. ['198.51.45.67'] [TTL=172800]
If both Dyn and R53 go down, it's exactly when you want a service like PagerDuty work without a hitch.
Spoiler: I'd bet my complete net worth against your assertion and give you incredible odds.
Golden rule: Fixing a DNS outage with actions that require DNS propagation = game over. You'd might as well hop in the car and start driving your content to people's homes.
I was giving a bare-minimum example of how this or (some other backup solution) should have already been setup and ready to be switched over.
DNS is bog-simple to serve and secure (provided you don't try to do the fancier stuff and just serve DNS records): it is basically like serving static HTML in terms of difficulty.
That a company would have a backup of all important sites/IP addresses locally available and ready to deploy on some other service, or even be built by hand via some quickly rented servers, is I think quite a reasonable thing to have. I guess it would also be simple to run on GCE and Azure as well, if you don't like the idea of dedicated servers.
I just wish it scaled to multiple cores :(
"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.
there is nothing stopping you from having Route53 and $others as NS records for your domains. You just have to make sure they stay consistent. Apparently from the linked discussion, there are people offering scripts and services to do just that.
githubstatus.com instead of status.github.com
You could even through the domain on a free DNS service.
That's one of the reasons I setup a git -> Route53 setup at https://dns-api.com/
From a technical perspective, if you're doing fancy DNS things like geo targetting, round robin though more A records than you'll return to a query, or healtchecks to fail out ips from your rotations, using multiple providers means they're likely to be out of sync, especially if the provider capabilities don't match. That may not be terrible, because some resolvers are going to cache DNS answers for way longer than the TTL and you have to deal with that anyway. You'll also have to think about what to do when an update applied successfully to one provider, but the second provider failed to apply the update.
From an organizational perspective, most enterprise DNS costs a bunch of money, with volume discounts, so paying for two services, each at half the volume, is going to be significantly more expensive than just one. And you have to deal with two enterprise sales teams bugging you to try their other products, asking for testimonials, etc, bleh.
Also, the enterprise DNS I shopped with all claimed they ran multiple distinct clusters, so they should be covered for software risks that come from shipping the same broken software to all servers and having them all fall over at the same time.
The only way that I could check to see if Github knew they were having problems was by searching Google for "github status", and then seeing from the embedded Twitter section in the results page that there was a tweet about having problems. Twitter also being down for me didn't help the situation either.
In this case, the servers (DNS server under attack at Dyn) that knows how to turn both www.github.com and status.github.com into an IP address were under attack and couldn't respond to a query. The only way to mitigate this would be to have a completely different domain (i.e. githubstatus.com) and host the DNS with a different company (i.e. not Dyn).
Anyway, for them to take the github.com nameservers out of the mix they would need a completely separate domain name; would you know to look there?
You can delegate subdomains to other providers, but the NS records are still present in the servers listed in the registrar. So, you'd already need multiple DNS providers.. And you wouldn't have been down. Just sayin. I'm not sure anyone rated a DNS provider of this status getting hit this hard or completely as high enough risk to go through the trouble.
It's easy enough to look at a system and point out all the things you depend on as being a risk. The harder part is deciding which risks are high enough priority to address instead of all the other work to be done.
https://www.dynstatus.com/ (using Route 53, at least today)
https://www.cloudflarestatus.com/ (using Dyn, ironically)
grep github ~/.ssh/known_hosts
sudo vim /etc/hosts
sudo killall -HUP mDNSResponder
ping github.comAlso probably the "hijacking top comment" part.
The other occurrence being here: https://news.ycombinator.com/item?id=12760156
192.30.253.112 github.com
but https://assets-cdn.github.com is failing
EDIT: Use 192.30.253.112 github.com 151.101.24.133 assets-cdn.github.com
or try 8.8.8.8 DNS
I briefly lost access to GitHub, but Twitter has been working fine every time I've checked. Posting status messages in multiple venues helps to ensure that even if one channel is down, people might be able to get status from another channel.
We're maintaining yellow status for the foreseeable future while the changes to our NS records propagate. If you have the ability to flush caches for your resolver, this may help restore access.
Latest status message: https://twitter.com/githubstatus/status/789565863649304576
192.30.253.113 github.com
151.101.32.133 assets-cdn.github.com
Edit; for profile pics include:
151.101.32.133 avatars0.githubusercontent.com
151.101.32.133 avatars1.githubusercontent.com
151.101.32.133 avatars2.githubusercontent.com
151.101.32.133 avatars3.githubusercontent.com
151.101.32.133 avatars4.githubusercontent.com
151.101.32.133 avatars5.githubusercontent.com[1] https://news.ycombinator.com/item?id=12762841
edit Of course this is if your local policy allows you to change this!
Edit: saw your other reply and looked it up myself, it's 23.235.33.133
pornhub.com:
Name Server: ns1.p44.dynect.net
Name Server: ns2.p44.dynect.net
Name Server: ns3.p44.dynect.net
Name Server: ns4.p44.dynect.net
Name Server: sdns3.ultradns.biz
Name Server: sdns3.ultradns.com
Name Server: sdns3.ultradns.net
Name Server: sdns3.ultradns.org
Name Server: PDNS196.ULTRADNS.ORG
Name Server: ARI.ALPHA.ARIDNS.NET.AU
Name Server: ARI.BETA.ARIDNS.NET.AU
Name Server: ARI.GAMMA.ARIDNS.NET.AU
Name Server: ARI.DELTA.ARIDNS.NET.AU
Name Server: PDNS196.ULTRADNS.NET
Name Server: PDNS196.ULTRADNS.COM
Name Server: PDNS196.ULTRADNS.BIZ
Name Server: PDNS196.ULTRADNS.INFO
Name Server: PDNS196.ULTRADNS.CO.UKpagerduty.com:
Name Server: NS-219.AWSDNS-27.COM
Name Server: NS-1198.AWSDNS-21.ORG
Name Server: NS-1569.AWSDNS-04.CO.UK
Name Server: NS-739.AWSDNS-28.NET
github.com:
Name Server: ns2.p16.dynect.net
Name Server: ns-1283.awsdns-32.org.
Name Server: ns-1707.awsdns-21.co.uk.
Name Server: ns-421.awsdns-52.com.
Name Server: ns1.p16.dynect.net
Name Server: ns4.p16.dynect.net
Name Server: ns3.p16.dynect.net
Name Server: ns-520.awsdns-01.net. nslookup
> server pdns196.ultradns.biz
Default server: pdns196.ultradns.biz
Address: 156.154.66.196#53
Default server: pdns196.ultradns.biz
Address: 2610:a1:1015::e8#53
> pornhub.com
Server: pdns196.ultradns.biz
Address: 156.154.66.196#53
Name: pornhub.com
Address: 31.192.120.36
twitter.com:
Name Server: NS1.P34.DYNECT.NET
Name Server: NS4.P34.DYNECT.NET
Name Server: NS2.P34.DYNECT.NET
Name Server: NS3.P34.DYNECT.NET
Name Server: pdns1.ultradns.net
Name Server: pdns6.ultradns.co.uk
Name Server: ns3.p31.dynect.net
Name Server: ns1.p31.dynect.net
Name Server: ns4.p31.dynect.net
Name Server: ns2.p31.dynect.nettwilio.com:
Name Server: ns3.dnsmadeeasy.com
Name Server: ns2.dnsmadeeasy.com
Name Server: ns4.dnsmadeeasy.com
Name Server: ns1.dnsmadeeasy.com
Name Server: ns0.dnsmadeeasy.comdigikey.com:
Name Server: cbru.br.ns.els-gms.att.net
Name Server: ns2.digikey.com
Name Server: cmtu.mt.ns.els-gms.att.net
Name Server: ns1.digikey.com1. Tried to download "Unknown Horizons" (game featured recently on Hacker News) binary, github-link doesn't work.
2. Think "Ok, might be an old link", google their github-repository, github appears down.
3. Try accessing github status website, is down.
4. Interested, try to visit github status twitter account, twitter is down.
Really weird experience, normally at least the second source of news on a downed website I try during an attack works.
"Popular tech site Hacker News reported many other sites were affected including Etsy, Spotify, Github, Soundcloud, and Heroku." -- http://fortune.com/2016/10/21/internet-outages/
$ dig @8.8.8.8 www.paypal.com
; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17925 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;www.paypal.com. IN A
;; Query time: 29 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Oct 21 12:35:33 2016 ;; MSG SIZE rcvd: 32
Also, I'm unable to resolve Twitter on TWC dns.
> This attack is mainly impacting US East and is impacting Managed DNS customers in this region.
I'm in Italy, using my provider's default DNSs (not Google) and I can't reach paypal.com, thenextweb, twitter, spotify etc either.
Shouldn't 8.8.8.8 query another name server if one fails to respond or takes too long?
EDIT: Is there an inherent flaw in how secondary records are queried? And as bhauer mentioned, is there a possibility to fall back to last known record?
Seems like it is affecting another regions now.
Github out, Etsy out, Paypal out, Twitter out, Soundcloud out, Crunchbase out, Heroku out, Spotify intermittent, Netflix only loads a white page with plaintext "who's watching" list and no functionality.
$ dig @8.8.8.8 www.paypal.com
; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40999
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.paypal.com. IN A
;; ANSWER SECTION:
www.paypal.com. 266 IN CNAME www.paypal.com.akadns.net.
www.paypal.com.akadns.net. 29 IN CNAME ppdirect.paypal.com.akadns.net.
ppdirect.paypal.com.akadns.net. 299 IN CNAME wlb.paypal.com.akadns.net.
wlb.paypal.com.akadns.net. 29 IN CNAME www.paypal.com.edgekey.net.
www.paypal.com.edgekey.net. 20 IN CNAME e3694.a.akamaiedge.net.
e3694.a.akamaiedge.net. 19 IN A 23.73.8.114
;; Query time: 146 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 21 13:05:48 2016
;; MSG SIZE rcvd: 198Squeaky wheels get grease and their sales team squeaks a lot.
I chose Dyn over Neustar (UltraDNS) when it was time to renew contracts because it was 60% cheaper, had a better latency, their support was great and the interfaces were clear.
Not a fanboy or anything, I really don't like how aggressively they hound me now (even though I have nothing to do with DNS for my current employer), but it's cheap and effective so it's not surprising people use them.
I bought that, and they've honored the deal. Admittedly it comes with limits that would make it useless for any large site, but it's just great for individuals.
Ironically, a quick search of my Gmail mailbox came up with this gem in the subject line from Dyn.
"Did you know the average cost of a single DDoS outage is $882K?"
$ dig +short ns amazon.com
ns1.p31.dynect.net. pdns1.ultradns.net. ns4.p31.dynect.net. pdns6.ultradns.co.uk. ns3.p31.dynect.net. ns2.p31.dynect.net.
They have been around a long time. For years they had a free product called DynDNS that would allow you get an A record for your dynamic IP at home.
So far twitter, etsy, soundcloud, spotify, github, pagerduty...crazy that this can even happen
$ host -t NS twitter.com
twitter.com name server ns4.p34.dynect.net.
twitter.com name server ns3.p34.dynect.net.
twitter.com name server ns2.p34.dynect.net.
twitter.com name server ns1.p34.dynect.net.
I'm not saying it's a good reason but it's a reason.
Krebs indicates in an update at the end that a source had heard rumors in criminal channels that an attack against Dyn was being planned.
Doug Madory's presentation was on the agenda for NANOG and so attackers would have had plenty of time to know about it.
From comments made on this and other similar posts in the past, I've gathered the following:
1) Malicious traffic often uses a spoofed IP address, which is detectable by ISPs. What if ISPs were not allowed to forward such traffic?
2) There is no way for a service to exert back pressure. What if there was? e.g. send a response indicating the request was malicious (or simply unwanted due to current traffic levels), and a router along the way would refuse to send follow up requests for some time. There is HTTP status code 429, but that is entirely dependent on a well-behaved client. I'm talking about something at the packet level, enforced by every hop along the way.
3) I believe it is suspected that a substantial portion of the traffic is from compromised IoT devices. What if IoT devices were required to continually pass some sort of a health check to make other HTTP requests? This could be enforced at the hardware/firmware level (much harder to change with malware), and, say, send a signature of the currently running binary (or binaries) to a remote server which gave the thumbs up/down.
- Attackers figure out something similar to the attack described above and can entice large amounts of users to visit a page that repeatedly fires at something like s3.aws.com or w/e, the user is unaware but they're essentially DDoSing s3.aws.com via attacker.com's webpage, and in point 2 they would be banned.
- DRDoS is similar to whats described above, but in point 2 it kind of stands alone as the biggest issue. it can be mitigated to a certain level by ISP's, but not entirely. (https://en.wikipedia.org/wiki/Denial-of-service_attack#Refle...) . Point 2 would actually help attackers poison DNS.
Even better, what if IoT devices were required to pass some health check to operate at all. This could be as simple as a verified boot plus a forcible reboot every now and then.
That seems believable.
> And as the Internet is decentralized you cannot order them to do anything.
...that doesn't. Being decentralized doesn't render them immune to regulation. If all major networks responsible for large scale peering were required not to pass on a certain type of traffic, it would be quite difficult to route around that. Yes, if only some did, this would be routed around.
This is worth reading. It has links to copies of the code and names the known control servers. Quite a bit is known now about how this thing works.
The bots talk to control servers and report servers. The attacker appears to communicate with the report servers over Tor.
This will cause supply-chain disruption for manufacturers using DigiKey for just-in-time supply.
(justdownforme.com says the site is down, but downforeveryoneorjustme.com says it's up. They're probably caching DNS locally.)
OpenDNS works because, as another poster notes, that, for better or worse, they don't strictly obey TTLs: https://news.ycombinator.com/item?id=12762429
If you can update your DNS to CNAME directly to the ELB behind it, it should at least make your site accessible.
edit: figured it out. What i did was do:
nslookup your-SSL-endpoint.herokussl.com
then you'll see the elb address.
Switch to the openDNS servers helpfully pointed out by someone above first...
208.67.222.222 208.67.220.220
(or specify dns server in the command)
; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: <id> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;<end-point>.herokussl.com. IN CNAME
;; Query time: 1226 msec ;; SERVER: <server>#53(<server>) ;; WHEN: Fri Oct 21 12:27:55 2016 ;; MSG SIZE rcvd: 44
the dig command does not work for me either...
=================================
nslookup iwate-2009.herokussl.com Server: 208.67.222.222 Address: 208.67.222.222#53
Non-authoritative answer: iwate-2009.herokussl.com canonical name = elb030330-152447250.us-east-1.elb.amazonaws.com. Name: elb030330-152447250.us-east-1.elb.amazonaws.com Address: 54.225.242.254 Name: elb030330-152447250.us-east-1.elb.amazonaws.com Address: 54.225.217.226 Name: elb030330-152447250.us-east-1.elb.amazonaws.com Address: 54.235.181.244
;; Got SERVFAIL reply from 10.17.100.2, trying next server Server: 10.17.100.2 Address: 10.17.100.2#53
server can't find <sslendpoint>.herokussl.com: NXDOMAIN
http://network-tools.com/nslook/Default.asp?domain=iwate-200...
I'm confused because of the use of "dyn dns", which to me means dns for hosts that don't have static ip addresses.
I'm actually surprised so many big-name sites rely on Dynect, which I hadn't heard of, but more importantly don't seem to use someone else's NS hosts as 2nd or 4th entries.
It may not be the proper action but this kind of soft-fail scenario (use the old DNS until you can contact the DNS servers and get new ones) is much better.
echo "nameserver 208.67.222.222" | sudo tee -a /etc/resolv.conf $ host -t NS us-east-1.amazonaws.com
us-east-1.amazonaws.com name server ns3.p31.dynect.net.
us-east-1.amazonaws.com name server ns1.p31.dynect.net.
us-east-1.amazonaws.com name server ns2.p31.dynect.net.
us-east-1.amazonaws.com name server ns4.p31.dynect.net.
$ host -t NS us-east-2.amazonaws.com
us-east-2.amazonaws.com name server u4.amazonaws.com.
us-east-2.amazonaws.com name server u6.amazonaws.com.
us-east-2.amazonaws.com name server u3.amazonaws.com.
us-east-2.amazonaws.com name server u2.amazonaws.com.
us-east-2.amazonaws.com name server u1.amazonaws.com.
us-east-2.amazonaws.com name server u5.amazonaws.com.
us-east-2.amazonaws.com name server ns2.p31.dynect.net.
us-east-2.amazonaws.com name server ns1.p31.dynect.net.
us-east-2.amazonaws.com name server pdns1.ultradns.net.
us-east-2.amazonaws.com name server pdns5.ultradns.info.
us-east-2.amazonaws.com name server ns3.p31.dynect.net.
us-east-2.amazonaws.com name server ns4.p31.dynect.net.
us-east-2.amazonaws.com name server pdns3.ultradns.org.
$ dig ns eu-west-1.amazonaws.com +short
ns3.p31.dynect.net.
ns1.p31.dynect.net.
ns4.p31.dynect.net.
ns2.p31.dynect.net.
$ dig ns eu-west-2.amazonaws.com +short
u6.amazonaws.com.
u5.amazonaws.com.
u2.amazonaws.com.
u4.amazonaws.com.
u1.amazonaws.com.
u3.amazonaws.com.
pdns1.ultradns.net.
pdns3.ultradns.org.
pdns5.ultradns.info.
ns2.p31.dynect.net.
ns1.p31.dynect.net.
ns4.p31.dynect.net.
ns3.p31.dynect.net.
us-east-1 is the oldest region and predates Route 53. Not adding extra DNS providers to the older regions is probably an oversight.
(The EC2 API team requests load balancers from a separate load balancer team. The load balancer team probably didn't exist as a separate team when some of these regions were created.)
$ host -t NS us-east-1.amazonaws.com
us-east-1.amazonaws.com name server pdns5.ultradns.info.
us-east-1.amazonaws.com name server ns3.p31.dynect.net.
us-east-1.amazonaws.com name server pdns1.ultradns.net.
us-east-1.amazonaws.com name server pdns3.ultradns.org.
us-east-1.amazonaws.com name server ns4.p31.dynect.net.
us-east-1.amazonaws.com name server ns1.p31.dynect.net.
us-east-1.amazonaws.com name server ns2.p31.dynect.net.
us-east-1.amazonaws.com name server u1.amazonaws.com.
us-east-1.amazonaws.com name server u2.amazonaws.com.
us-east-1.amazonaws.com name server u3.amazonaws.com.
us-east-1.amazonaws.com name server u4.amazonaws.com.
us-east-1.amazonaws.com name server u5.amazonaws.com.
us-east-1.amazonaws.com name server u6.amazonaws.com.Don't confuse regions with availability zones. (Though in this case, the availability zones don't help...)
6:36 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.
It happened to be at the same time I was getting things configured to connect to a new VPN that I hadn't used before for the first time. Until about 7am today my home network was a 10.0.0.0/8 network. VPN kept bombing in the last phase of connecting and I couldn't figure out why, so I thought it was an IP conflict with my internal network range.
So naturally, I then went into my router and changed my subnet for my entire home network to the more common 192.168.1.0/24 range to see if it'd help. It didn't. Until suddenly VPN "just worked" -- which makes me wonder if I needed to change my network at all to begin with.
Then I started experiencing all sorts of weird issues where the Internet seemed to disappear from one minute until the next.
Then I hit IRC when things finally stabilized and see "Did you hear about Dyn?".
My reaction: wut.
TL;DR: I rearchitected my home network at 7am for no reason.
Also what is happening is a cascade effect, where a 3rd party being down effects others.
I'm a fan of Route53, too.
But can we say that it weathered the attack? Or was it just lucky that its systems weren't targeted?
I was wondering that too
I've seen them be hit by dDos attacks in the past, but never had any significant impact.
(I wrap Route53 and handle storing DNS records in a git repository over at https://dns-api.com/ Adding support for other backends is my current priority to allow more redundancy.)
Or would that not actually solve this particular scenario?
ic.ac.uk. 45665 IN NS ns1.ic.ac.uk.
ic.ac.uk. 45665 IN NS ns2.ic.ac.uk.
ic.ac.uk. 45665 IN NS ns0.ic.ac.uk.
ic.ac.uk. 45665 IN NS authdns1.csx.cam.ac.uk.
>> We are seeing a widespread DNS issue affecting connections to our services both internally and externally.
Max record cache time is 86400s (24h), so if the attackers can keep it down for 24h then google will have to have custom instructions in place (or cache more aggressively than the RFC allows)
In addition, Google Public DNS engineers have proposed a technical solution called EDNS Client Subnet. This proposal allows resolvers to pass in part of the client's IP address (the first 24/64 bits or less for IPv4/IPv6 respectively) as the source IP in the DNS message, so that name servers can return optimized results based on the user's location rather than that of the resolver. To date, we have deployed an implementation of the proposal for many large CDNs (including Akamai) and Google properties. The majority of geo-sensitive domain names are already covered.
#8:07 AM 10/21/2016
199.16.156.70 twitter.com
104.244.43.231 abs.twimg.com
104.244.43.231 pbs.twimg.com
192.30.253.113 github.com
151.101.24.133 assets-cdn.github.com
Recent IoT-based Attacks: What Is the Impact On Managed DNS Operators? - http://hub.dyn.com/traffic-management/recent-iot-based-attac...
It's a good piece about how IoT-based DDoS attacks are carried out. And now Dyn has the answer...
HN thread about that article at: https://news.ycombinator.com/item?id=12764650
Only 2 of the points in the US are affected on https://www.whatsmydns.net/ for the domains we've got on Dyn - same for Twitter etc
Krebs on Security: https://news.ycombinator.com/item?id=12761859
NY Times: https://news.ycombinator.com/item?id=12765652
But it should be "obvious" if your users report "Server not found" vs. "Cannot connect" or "Page not found" style errors.
Query for the root domain, without any subdomains like www. That is, you need to check the "zone apex," the shortest name purchased from a registrar and potentially delegated to Dyn. Look for dynect.net in the list of authoritative name servers.
But then, what is China or Russia going to get out of doing something like this? It isn't going to change anything. Hillary is the next president regardless. Hell, even if no votes could be counted I am sure the Supreme Court wouldn't have a problem calling it for her.
So to me the idea that China and Russia is doing this for political reasons doesn't make any sense.
Who would have thought adding a spof to your infrastructure would ever be a problem?
No GitHub, well, it's gonna be a fun Friday...
192.30.253.113 github.com
For example, most open source projects.
But even outside of that, we use github for issue tracking, the new project management kanban stuff, a CI server, reading documentation (which is offline, but the online versions are nicer on the eyes), and a ton more. Not to mention that StackOverflow and other discussion forums tend to be used by many.
Yes you can host all of that locally, but we don't have a few hundred thousand a year to spend on some sysadmins to maintain all of that, and we don't have the time or money to run the machines, vet the software, and keep it up more reliably than github does for next to nothing.
I disagree with that. If you have a server, you need a sysadmin. End of story.
Who is going to secure the system and setup ssh keys? who is going to run updates? who is going to monitor for security issues? who is going to run backups? who is going to secure those backups? who is going to oversee the installation of the network, the battery backups, the racks, the server hardware, etc... Who will swap out bad disks? Who will recover the system when it goes down? Who is going to double the hardware and setup high availability (remember, you are competing with github for uptime here)? And god help you if you have one guy that does all of this. What happens if he gets hit by a bus?
An on-prem server isn't a "backup", it's a liability. And without the resources to maintain it, it's going to become a nightmare. I've been there, and I won't ever do it again.
I'm either going to pay to do it right, or give it to someone who will. And if that means a few hours of downtime every year or so, then that's a wonderful tradeoff for me.
> It does not take long time or resources to download all of the libraries, with corresponding docs to a local server, or even your laptop. It is not complicated to have all of the new issues sent to an email to have a version of them available at all times.
Luckily github (and alternatives) provide all of that. It sends us emails and slack messages on everything, so if it's down, we can still read, and we all have our local repos. But reading is different than working.
If self-hosted, somewhere, you could still be screwed by having Dyn as your DNS provider.
If dev-machine-hosted, then uh, your issue tracker is no longer an issue tracker. Your build server is not a build server. All the services besides Git are not meant to operate offline in a decentralized/distributed fashion.
Library documentation, sure, that could be local. Otherwise, your assertion that all of this tools infrastructure can somehow be replicated, easily, in a way that makes the difference between working online or offline effectively zero, is nonsense.
Second, pointing to a new machine is as simple as updating IP in your hosts file, or dns server.
Third, you can use vmware or any other virtualization stack to replicate your infrastructure locally. In fact that's the best way to build things - create virtual network, use it for testing, troubleshooting and debugging, and deploy only when everything is working.
All I'm saying is that if you're company is making any kind of money, and your development environment depends 100% on online services, you're doing it wrong.
If that's the case, i think you're misguided : imho, the internet as it was designed was conceived so that everyone has its little self-hosted thing, with dev-machine just for the purpose of, well, test and dev, with the latter goal of it being self-hosted.
Just look how email is technically designed and how it was meant to work, and we use it now, relying mostly on Gmail or Outlook, or worse, using Facebook for emails : we put all our eggs in the same basket.
Option A) Spend no money and experience an outage maybe once a year, if that. And the problem works itself out.
Option B) Spend money and gain technical debt to avoid a problem that happens maybe once a year, if that.
Which one would you pick? I mean, maybe if everything you have is closed-source or you are guaranteeing 99.9% uptime to your customers, perhaps option B makes sense. Otherwise, the choice seems fairly obvious to me.
I work in IT infrastructure and I see attacks literally every day, moreover, most people just setup a quick LAMP or MEAN stack to prove their concept, and then they leave it like that, so most of the time, no, the problem don't just "work itself out".
Yup. But the whole point of GitHub is to make you dependent on GitHub.
They've been slightly successful.
Go to http://whois.icann.org/en and enter your domain name and see what info is public about you. If all your info is public, you may want to see if your registrar offers "private" registration where your info does not appear in WHOIS.
Attacks at this scale can bring a significant part of Internet down. The economic affect can be just as bad as a war.
Let's hold the ISPs financially liable for the harmful traffic that comes from their network. If a client reports a harmful IP to the ISP, every bit of subsequent traffic sent from that IP to this client carries a penalty.
Yeah, I know, routing tables are small, yada yada. If we put thumbscrews to the ISPs they will find a way to block a few thousands IPs of the typical botnet, even it requires buying new switches from Cisco & co.
Incentives drive behavior.
You wouldn't allow car manufacturers to sell cars with faulty airbags, why do we allow device manufacturers to provide plentiful firepower for bad actors?
How would you even start chasing a manufacturer of a cheap IP cam from China? How many of them can you chase at once?
Then when I went to push to github out of fear my computer was about to soil itself, that failed too, and I noticed the outage.
Does anyone know if the above errors could be related to the outage? I'm using vim inside tmux with zsh as my shell. Maybe zsh does some kind of communication with gh while running?
I restarted my computer and it's still happening
Are you using some oh-my-zsh github plugin by any chance?
https://github.com/robbyrussell/oh-my-zsh/wiki/Plugins#githu...
oh-my-zsh is, imho, way overengineered and bloated. Leads to all sorts of issues like the one you're encountering there.
I would recommend sticking to a plain zshrc file that you can read, edit and fully understand.
The one I wrote and am using day to day is available here, with documentation: https://github.com/jleclanche/dotfiles
77.88.8.8
77.88.8.1
https://gist.github.com/agh/4e20df0d2d3bfa189477569b77f72e24
www.ft.com is unreachable for example.
They run some sort of web server since most devices provide some web interface, so clearly there's a port open which could be hit if the IP is know, and with the shoddy security in these devices I'd wonder if their local (likely low performance) hardware would be susceptible to something as simple as a ping flood attack.
If one DNS server is down, use the cached result or another server.
DNS is some of the most distributable, cachable data I can imagine.
I used to work for a DNS/DDoS provider, and this was a very real problem. Leave the PoPs that are being affected out, or risk overloading the other PoPs by overloading real traffic.
Before moving the other traffic, you also have to worry about blocking the DDoS traffic otherwise you're just redirecting them to the other PoPs. Mitigating DDoS attacks are not fun, and hard to block.
This was in eu-west-1, but it coincided with a bunch of other systems in the organisation having problems at the same time.
Additionally CloudWatch logs seemed to be completely broken for about 30 minutes on the Amazon Console.
https://www.reddit.com/r/sysadmin/comments/58o5mp/dyn_dns_dd...
dig @8.8.4.4 github.com +short
However using my routers/dhcp/ISP to set my DNS server, I am able to get answers:
dig github.com +short
192.30.253.112dig +trace github.com
[1] https://twitter.com/AlexJamesFitz/status/789562789920636928
[2] https://krebsonsecurity.com/2016/10/source-code-for-iot-botn...
tl;dr: a substantial part of this attack is a botnet of IoT devices.
If this kind of attacking does escalate, wouldn't it be possible to simply cut off requests from outside the United States at the points of entry? Basically, turning the US into an intranet?
"Oh, maybe its our shitty ISP screwing up everything again."
No, it's in a bigger scale.
192.30.253.113 github.com
151.101.4.133 assets-cdn.github.com
Second, Cargo only depends on GitHub for the index. For more: http://integer32.com/2016/10/08/bare-minimum-crates-io-mirro...
That includes a link to a mirror run by integer32.
> ....point your machine or router's DNS to use opendns resolvers instead of your regular ones: 208.67.222.222 and 208.67.220.220
GET https://art-s.nflximg.net net::ERR_NAME_RESOLUTION_FAILED
GET https://assets.nflxext.com net::ERR_NAME_RESOLUTION_FAILEDMaybe this is their strategy: we break it, you buy it ;)
https://help.dyn.com/internet-guide-setup/
If you can't load that page, the public DNS servers are: 216.146.35.35, 216.146.36.36
OpenDNS - recursive DNS
Cloudflare (DNS only) - authoritative DNS
Both services are free and distributed across the world.
The takeaway here should be to use multiple authoritative DNS providers, not a single (even if better) one.
Is it possible the government could force a DNS provider to pretend to fall victim to a DDoS attack, as a form of a false flag cyber attack?
Kids.
Cisco could step up to the plate here. And no, I'm not talking about firewalls. We need newer ICMP type packets to create this back-pressure, so that we can stop floods like this.
What is this type of attack? TCP/UDP/ICMP has no notion of of a compromised host. Or even that it was crafted packet.
Back pressure already exists in TCP, see slow-start and window sizes, flow control is part of the "control part". When a router's porst buffers are full the router drops the packets on the floor. It does not do further processing of those packets.
What would ICMP do here? If I have a million compromised hosts and each sends a single SYN packet towards a destination host, how would ICMP help?
">Cisco could step up to the plate here" What would Cisco do? Cisco doesn't control the ICMP protocol.
I think you are not understanding ICMP. The job of ICMP is to report error conditions. ICMP serves as a helper to IP which is itself unreliable and has no form of error control or checking. A router or host being overrun is not a network error condition it is a resource condition. No ICMP type is ever going to be able to stop a host from originating UDP/TCP/ICMP towards a destination. Even if it could you would just overwhelm it in the outbound direction by replying to potentially millions of hosts.
A better analogy would be 'mocked cyber mass protests' seeing as how no infrastructure will need to be rebuilt after this passes.
Critical infrastructure MUST NOT rely on PUBLIC networks like the Internet. It's a TERRIBLE idea. If you're working on anything actually critical, build your own fucking ISOLATED network with your own fucking cables.
The USG is not responsible for a corporation that runs DNS services.