Taking Over DigitalOcean Domains via a Lax Domain Import System
thehackerblog.com
thehackerblog.com
I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later.
Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.
I had to pay 5€ to even be able to add the 100$ credit from the GitHub students pack to my account (for "verification purposes"), and then they – illegally – delete it just like that? (I never got to use any of it)
DO is one of the shadiest hosters I know.
For what it's worth, we posted on our blog about just this. https://www.digitalocean.com/company/blog/details-on-expirin...
I got the ToS change notification, and merely 2 weeks later my credit was invalidated.
I had had the credit on my account for over 2 years by that time, but hadn’t used it.
And I had opened a support ticket, and was advised to take legal action if I believed it was invalid or wanted it back.
No thanks, I’ve had enough.
At no point should anyone be advising you to take legal action to retain your credit, and I would be grateful for the opportunity to review the ticket in question.
I agree with you completely. That is precisely why I refer to it as a communication failure. I had a year to ask "Why didn't we email about this?" I didn't ask that question. I failed. I'm sorry.
I understand that you're not interested in helping, but my offer stands, and I appreciate you and your feedback. We're all just people here trying to do what we think is right. I know I fall short, I make mistakes, but all I want is to own them and make them right.
Why did I not receive any notification? A change in contract of that magnitude should have led to me being directly notified per email or other communication methods, so that I’d actually have a chance at still using the credit.
This way, you notified me right when the credit ran out, which was extremely shady, and I’ve met hundreds of others with the same issue in IRC when you invalidated it.
I can’t and won’t be able to assist you with an investigation to improve your public image, though, as I’m quite busy at the moment, and my time is better spent contributing to open source projects than discussing the wrongs of a hoster that won’t change anyway on HN.
>A change in contract of that magnitude should have led to me being directly notified per email
It seems to me the credit is a form of charity for students, and you had not bothered to redeem it for 2 years. Now, you seem to be implying that DO is acting in an evil manner for only alerting you to the change via the dashboard (which the DO rep in this thread admitted was poor execution on their part). I don't think you are being charitable, and I think the underlying assumption you are making almost sounds conspiratorial.
I'm not a DO customer, and don't have any affiliation.
The credit was invalidated 3 months before that.
With the legal 3 year validity period of promotional credit and coupons that German law guarantees, I had expected being able to do that.
https://www.digitalocean.com/company/blog/details-on-expirin...
That left me just 2 weeks.
And what’s illegal is that in Germany, promotional credit or coupons given without a time limit is valid for at least 3 years.
So you confirm that a set of name server belongs to a user by verifying that its domain record has those name servers configured. You then don't allow another user to control the same name servers for that domain as long as the domain record still points to them. New users can then freely create new name servers but the domain still uses the servers assigned to by the domain owner.
What DO does is that it allows any user to control the name server for a domain if the previous owner gives up control of them. It should only do that, once the domain records start pointing anywhere else.
I would totally support the ACME spec being expanded to cover this, and various other domain verification related issues
I asked them to, at the absolute least, send an email notification to the prior-CloudFlare owner letting them know that the domain "your CF account used to control is now being controlled by a new CF account". Better yet, implement a domain ownership validation scheme.
They told us that they wouldn't be making any changes.
FWIW, on CloudFlare what happened to us was: we were moving registrars for ~100 domains, from GoDaddy to Route53. During this transition, the NS for the domains temporarily became blank; at this point CF automatically removed the domains from our CF account. The NS were then re-added to the domains on the Route53 side (<4 hours of 'no nameserver' time).
Apparently there are people out there that are looking for domains that are pointed to CF and then attempting to add them to their own CF account (automated I'm sure) -- which CF lets them do without any verification once they've been auto-removed from your [the original CF account] account.
Interestingly, the original account must be still stored in their system with the domain because we were able to re-add the domain to our original CF account without any verification; effectively "stealing the domains back" to our CF account, away from the thieve's CF account.
In this case, the "attackers" (perhaps more appropriate, I call them 'malicious actors') were able to commandeer ~100 of our domains for ~2 months, for free; they redirected them to Russian websites, torrent sites, affiliate sites, etc.
Again, this is being actively exploited on CloudFlare, at the direct expense of CF customers -- but, according to CF, it's not an issue...?!
According to CloudFlare, they are are a reverse proxy, and they are not responsible for anything. This has been their response to every issue that I've tried to bring up with them over any channel, including here on HN.
CloudFlare just doesn't care.
The policy on that web page has two egregious issues.
1) It does not have any provisions for SPAM. I regularly get e-mail SPAM that has CloudFlare-protected links. That abuse page does not even apply. Effectively, CloudFlare offers spammers a 'pink contract'.
2) CloudFlare has a reputation for ignoring abuse, and that page effectively says nothing about whether abuse will be stopped; that page only offers to transmit the personal information of someone who reports abuse off to an abuser. This is not a theoretical; this has happened in the past, and the personal information of someone who reported a site that contained child pornography was then posted all over the net for its users to begin harassment with.
So if you find something abusive or malicious, your best bet really is not to report it. CloudFlare won't act, other than to put you at risk.
I was basically told "you will be wasting your time"...
They just don't want to be responsible, and unlike any other large network service provider that I sent abuse reports to, they just try and deflect and keep servicing spamming and phishing operations instead of booting them from their systems.
Definitely not the case.
I work at CloudFlare, not in DNS or on this code, and have mentioned this incident in the all-company chat. There is a healthy conversation happening there and it turns out a fix was already in the works for the underlying issue.
adanto6840 has supplied the support ticket number (thank you), and this specific incident is also being reviewed.
Great find / writeup.
They took almost twenty thousand, sent all the requests to their own server and logged them.
That's surely not your first step.
And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.
Well they were still being accessed by real people.
> And if DO was really concerned about the damage then they would have removed the added A records, but they didn't, they banned the author and continued to let all traffic go to his server.
Both seem reasonable. And it does sound like the security team are actually doing something about it.
> I reached out to DigitalOcean’s team to see if they can assist in deleting the domains from my account (sadly leaving them vulnerable again) or sinkholing the DNS to 127.0.0.1. I received a very helpful response from someone on the security team and it appears they will look into it.
I assume these things happened from different departments. Banning an account because you saw it make 20k requests to your API adding domains seems pretty reasonable, and it was a few hours before they reached out. If you saw that activity, would you ban the account or leave it open hoping that they'd be doing something nice and reach out?
If I was a domain reseller, adding my 20k domains to digital ocean just to get banned without warning, explaination or option for reconsideration I would be rightfully upset. If they didn't want people adding large numbers of domains they could just have limited the feature instead of banning people who reach some arbitrary threshold.
If on the other hand the department that executed the ban knew that the registrations weren't made by the domain owners, they should be discussing such a huge incident with the security team. That discussion would naturally lead to them knowing about the specifics of this case, unless this case wasn't widely shared in the security team.
So the options are:
1. Digital Ocean bans legitimate customers without warning or option for reconsideration; for no obvious reason
2. Big security incidents don't get reported to the security team
3. The responsible people thought that this was not a big security incident
4. This incident wasn't discussed in the security team
5. They knowingly banned a white hat hacker (who may or may not have gone too far)
Of all those options, the last one is by far the one that looks best for Digital Ocean.
Neither of us really know what's happened here, but the author at least thinks DOs actions were justified so I'm reasonably happy to leave it at that.
I would be careful about doing something like this on a large scale, I would actually be surprised if this doesn't technically violate some laws. Please be careful, you don't want to end up trying to explain nameservers to a judge. [edit - and the rest of us don't want that either, even just selfishly I would like as many security researchers practicing their craft as possible, but also I don't want people being punished for trying to do the right thing]
I just wanted to let you know that I really appreciate your feedback, as well as the feedback from the other commenters here.
I understand that many here are concerned that banning the account seems, from this perspective, to have been an unjustified action. I do believe that there is a bit of a misunderstanding on the timeline of events here, as well as the source of the decision. To be clear, Cash supported the decision that I had made to ban the account in question, and there had been no communication between us and Matthew at this point. We began receiving a significant number of support requests to remove domains from this account, and I authorized the shutting down of this account as it was clear to me what was happening. I have been working with our engineers to see to the removal of the domains from the account as well.
I apologize if our actions seemed at any point rude or inappropriate, it was definitely not my intention. I want nothing more than to look out for the safety and wellbeing of our customers, and I chose what I believed to be the best action. I do want you to know that if I was aware that a security researcher had been working on testing a theory, I might have acted differently. That can, however, impact the reason behind a white hat test. You generally want the company to see you as normal user, so that you can see how they act in return. We do shut down users who are intentionally causing problems for other users, and I do think that was made evident here.
I do understand that opening a line of communication with Matthew may have been appropriate, and I consider that valuable feedback moving forward.
<3 Jarland
At the very least I'm happy that some people have seen this more as a study of this pattern of functionality being an issue instead of this being only a DigitalOcean problem. I've seen (and have been reached out by) multiple people realizing this affects their company as well which has been awesome to watch.
I don't think you can make a proper decision where you're only looking out for protection, or only looking out for convenience, or only looking out for expected behavior. I think you have to mesh all of these items together and make a change that addresses each item, and I don't think that's necessarily a one day discussion. Certainly security does come at a cost of convenience, and that is okay, but it is important not to toss convenience aside as something not worthy of consideration.
So yeah not trying to be vague, my position is not with engineering or security but with the support team. I do think we need to talk about this, and conversations are taking place, but I don't honestly have the ability to say "Yes we're going to implement _____ within ____ days" or something like that. At least not today.
Let's remember Linode offers 2x the RAM.
I'm a long-time Linode customer and use them for all servers that run important services. I use DO for the less important stuff. But Linode has had a not-so-stellar record of data breaches.
Gray hat. (Still doesn't justify the idiotic response from DO security team)
Took a few emails to Cloudflare support to resolve this one. They also didn't seem to care much about the security implications when I questioned them about it.
So this is far from a DO-specific issue...
Sort of hard to call it a vulnerability on DO's part though - more of an issue with the admins. I think most DNS services operate in this way, really, route53 may be the exception, not the rule.
This is an absolutely terrible response from DO. If I had anything hosted here, I'd move away ASAP. Seriously, do it.
Where are you going to run off to? How is their security better over there? How many hours of work does that involve?
I think you're looking more kindly on their security practices than most folks in the security world would.
DO knows people can do this, but they don't want people to do it.
Remembering 2 recurring DNS servers is easier for bulk management than a bunch of different ones.
You don't test services like that, it can negatively affect other users.
Response wasn't perfect but it was reasonable.
I'm sorry, "We aware that we make it easy for about 20k domains to be directed to a malicious host, but we're not going to do anything about it" is reasonable?
But of course, it's because Matt "was messing around with things he shouldn't be". It's all solved - we just need everyone to stop doing things that DO "don't want people to do".
If the domain is added to the account there is no PoC, it's only for domains that have been removed from accounts but still have the nameserver values(meaning the domain is not being used at this point, there's no zone file if it isn't added to an account).
So this is mostly only going to affect currently derelict domains. I'm not saying it isn't something to worry about, but I do think it's a reasonable solution.
If I remove my domain from Digital Ocean, it's my responsibility to then go to the registrar and point the registrar away from DigitalOcean's nameservers. (I own the domain, so I'm the only one the registrar allows to do this. Digital Ocean cannot do this.)
Now, your suggestion is that Digital Ocean goes and verifies that I'm the one who owns that domain. But how would they do this (legitimate question)? I imagine manual verification of ownership of every domain upon creation isn't feasible for their scale. Digital Ocean could query DNS, and see NS records pointing to Digital Ocean, but this only tells them someone configured the nameservers for that domain - it doesn't imply ownership. Digital Ocean can check Whois for the owner of the domain. Checking Whois might work for many cases, but at least some registrars have the option of obscuring Whois data.
It seems simpler to put the onus of security on the owner of the domain. I should cleanup my registrar's NS records before removing my domain from Digital Ocean to ensure nobody hijacks it. I would be satisfied as long as Digital Ocean maintained a simple eviction policy (I don't know if they do) as a way for legitimate owners to add their domains to Digital Ocean's nameservers.
(Obviously when I say somebody else's NS I mean a NS they have zero reason to think would respond with correct records. Obviously not talking about outsourcing DNS hosting.)
I'd much rather know about this vulnerability via a researcher than when a high profile company loses control of their accounts / domains.
Those aren't mutually exclusive. It's certainly possible to be broken by design.
> This would only happen if it's something you're not using anymore
From the writeup it seems like it would also happen if you registered a new domain and assigned the DO name-servers but didn't immediately point it to something.
However it's the way I've always done it, because otherwise somebody else could add the domain first... it just made sense to me I guess.
[1] https://serverpilot.io/community/articles/how-to-configure-d...
Was there a realization into how legitimate users may be affected by this action? Was there a plan to remove those domains from their account after making and disclosing their proof of concept?
Why not stop at 10 or 20, and then alert DO to the findings?
20 thousand was unnecessary.
It was my plan to delete the domains (or at least null route them so others couldn't take them over with more malicious intent). However my account was banned before I could do so.
Remember, it's just another nerd out there somewhere who receives your report. Sometimes they are super stoked to have it and will be very responsive. This can be more satisfying than a public disclosure without first contacting the company.
How would it have ended if you hadn't reached out to one of their security people after being banned? They'd have given you the traffic, locked your account, and congratulated themselves on a job well done?
Put another way, the domains were entirely unresolved and offline. Then they weren't. If anything, this is a nice lesson about keeping your zone and delegations clean, and I'm glad I read it. Nobody got hacked, nobody lost traffic, nobody was impacted. If there was sensitive traffic going to a non-resolving domain, I have more questions than answers. I agree adding the full set was probably a bit much, but you can make that point without misplaced concern for alleged harm.
I'm not impressed with signing up for HN to hit someone like this and your far worse and flagged followup, particularly since it really looks like astroturfing. I guess take solace that the OP pretty much guaranteed he won't get the zones again, since they're for research.
(Nice ninja edit and deletion.)
Adding 20k domains to your nameservers at once might cause issues in resolution while it catches up with itself which will affect other users using the DNS service. It shouldn't on DO's scale, but it might. This is why the whole thing isn't a "white hat" situation, the author tested without thinking of potential consequences.
Someone can easily block you from using S3 static website hosting by adding a bucket with your domain name before you do. Also if you delete a bucket and do not change your DNS, someone can recreate the bucket and will be serving files from your domain.
This way you are not limited by bucket names and you also avoid any SSL validation errors.
Note: you can set Cloudfront's TTL to 0 if you don't need any caching.
However you are going to pay for Cloudfront requests in addition to S3's. In the vast majority of cases, it's insignificant compared to the savings you can achieve on the bandwidth side. (And at a certain level you can ask AWS to waive CF request fees entirely).
That's not correct. The S3 bucket name is always prefixed. The format is: bucketname.region.amazonaws.com.
https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosti...
To clarify, you're going to have to add a DNS record either way. Doesn't matter what you call your bucket. And no, you don't need to put CloudFlare in front of it for this.
While his intentions might have been good (and I expect that they were!), that kind of behavior isn't.
If he was operating responsibly, he would have applied it to a domain he controlled and provided that as a proof of concept. Instead, he ganked twenty thousand domains. That is at best irresponsible and at worst malicious and DigitalOcean (who I am no fan of, for what it's worth) has no obligation to figure out, or even care, which is which before showing him the door.
Which ones?
A few examples:
- Whenever a new release of OpenBSD comes out, I send a request for them to add the new install media, and they make it available ASAP.
- I switched from OpenVZ to KVM after a couple months and having paid for a year of the OpenVZ service, and they put my remaining credit toward the new service.
- I completely hosed my machine one time and had them restore from backup. Took a while because they wanted to be double-extra-sure of what I wanted, but it was overall a quick and painless process.
I don't know how things will change as they grow, but right now they just seem like a good company, run by people who care about the quality of their product and the satisfaction of their customers.
The difference is I didn't exploit 20 thousand domains to make flashy headlines and prove a point about something that isn't even a serious bug.
I'm less inclined to believe the latter. I'm looking forward to transferring my domains on DO to elsewhere tonight.
I'd be happy to share bug tickets with anyone who isn't on some silly chill hunting crusade, for sure, despite my trollish throwaway name.
I'm doing math on the throwaways that are oddly attracted to this thread. You are making it very obvious that you are almost certainly a DigitalOcean employee across the two throwaways you've created so far, and that's giving me a whole lot of pause on DigitalOcean that I didn't have from reading the incident itself. If you're an employee or, less likely, a superfan, are you sure this is the type of sustained attack you want to levy? It's not making anybody look good.
Probably not.
Most admins don't think about a complete TLD failure. Amazon did.
Agree
>> Most admins don't think about a complete TLD failure. Amazon did.
I think companies such as Google or Facebook did think that before, but I am not sure why they didn't follow this trick.
If the service doesn't understand the issue at all, then when you explain that they're your domains, then they'll probably just tell you it's working as intended and that users should be able to add their own domains.
When they fail to understand that then feel free to go ahead and pick one that's not yours to prove the point. One though, not 20 thousand.
I think this was his mistake.
The security team at DigitalOcean has been working to ensure that DO is a safe place for security researchers to identify issues on the Internet as well as at DigitalOcean - security is in everyone's interest. We encourage researchers to contact us when they want to use our platform for this type of work specifically so that we can avoid the types of pain that Matthew encountered while doing his experimentation.
Feel free to reach out to security@digitalocean.com and we will be happy to help.
Nick, DigitalOcean Security Director
Now I'm in prison because of this so it's really hard for me to put it back."
The article writer is an idiot. He deliberately stole accounts because he could. Just because he then decided to blame the provider because he was able to do this does't make it any more defensible.
If I mug you in the street, should I then post that because I was able to do so it's all your fault? No. I'd go to prison....
A better comparison is removing 10 people's lunch money from their school lockers, maybe due to a careless sequential scheme of creating combinations, and giving the money back. And doing this before talking to the principal or any teacher.
Either way, he still should have contacted them before.
He just configured a bunch of deleted/unconfigured domains to point them to a blank web page running on his own server. The point is that he could have done all sorts of nefarious things with that redirection but he didn't. He made a harmless change to demonstrate the vulnerability. That's what white hats do. It's what they're supposed to do. We should thank him for his efforts not lambaste him for actually doing something about the problem.
All of those domain owners are free to change their nameservers at any time.
The first person that replied looks like he just skim read your email or didn't understand the fact you had sinkholed a lot of traffic.
It sounds to me like he read your email about the vulnerability, dismissed it as "on the backlog" and skipped the bit about the fact you had sinkholed lots of traffic.
We only have one side of the story though, so who knows. Lessons can be learnt.
http://security.stackexchange.com/questions/49612/how-does-d...
At first I thought it was an issue where DO was using their own DNS servers but letting you add domains. So if Microsoft.com wasn't registered, you'd register it then all DO customer traffic would get your DNS records.
Some telcos have this. Sign up, say you're porting a number. Provide number of local bank and invalid port info. Account gets set up, telco routes their own calls to you. Port is rejected and customer service is asking for info. All while you are getting the bank's calls and forwarding them. Perfect MITM.
Luckily I was able to get my number back, after many calls to customer support (they managed to undo the porting when I wanted to check how it was going) and a couple of weeks of waiting.
EG if foo.com is a working site on your DNS provider, try creating a zonefile for bar.foo.com and see if you can create an A record to point to your own server.
This used to be something shared web hosting services running CPanel/WHM were particularly susceptible to. Clearly, the risks here are both phishing/identity and cookie credential stealing.
It's not enough to just come up with the custom nameservers. In order to use them in most TLDs they also need to be "registered" with the registry that operates the TLD.
So let's say you have myDNSdomain.com. You get a new customer who owns NewCustomer.com and wants to you your DNS, so you create these nameservers for them:
ns237.myDNSdomain.com ns2323.myDNSdomain.com
In order for your new customer to be able to use those on their NewCustomer.com domain, you will need to go to your registrar and set up these nameservers. The registrar will then create the corresponding nameserver records with Verisign, the registry. Only then, the customer will be able to use the nameservers on his domain.
I feel like this question yields better context to ethical arguments because it makes us aware of the cognitive biases and view things from a more abstract perspective..
EDIT: Is there a way to include plain asterisks in HN posts?
The only defence AWS has against this type of attack is the random (?) grouping of four different NS?
Some companies do seem to prefer to learn about vulnerabilities from pastebin database dump.