MarkMonitor left 60k domains for the taking
ian.sh
ian.sh
And then:
“ MarkMonitor does not have a way of disclosing security issues, which inhibited reporting this to them in a timely manner. They have not responded to any of our communications.”
Should anyone be surprised that a company that claims to not make mistakes, but has no way to report vulnerabilities gets their vulnerability on the front page of HN? (And this is a good case scenario, as opposed to their customers being hacked: which could have happened by an attacker claiming some of these domains)
Let’s hope they make at least the change of putting a bug bounty program in place.
But that was for domains we were actively using. I don't even remember if we had a parking page on the domains we only had to sit on; if we did, while it wouldn't have been great, it also wouldn't have been a big deal for them to show something attacker controlled for a little while. Parked domains weren't in active use, didn't have links going to them, and probably shouldn't seem credible to others.
I think we might have set up the parked domains to just redirect to our main website, but yeah.
This is understating the risk. If an attacker obtains control over a website for even a brief period of time, they can go to a certificate authority and obtain a domain validation for the domain (including all subdomains) that's valid for one year. At the end of that year, they can use the domain validation to obtain an SSL certificate that's valid for up to one year, even if they no longer control the website. This means that following a brief takeover, the domain continues to be vulnerable to attack for two years, which would be a problem if a parked domain is transitioned to active use during that time.
[1] https://datatracker.ietf.org/doc/html/rfc6962#section-4.1
[2] https://datatracker.ietf.org/doc/html/rfc6962#section-3.3
It's a condition for your certificates to work in some popular browsers (notably Chrome and Safari) but not a policy requirement.
This is on purpose. Google themselves for example will get a certificate issued, say, today, for shiny-new-google-product.example and then only at the last moment do they log that certificate when spinning up the web site https://shiny-new-google-product.example/ to launch shiny-new-google-product -- you can't find out about it in the CT logs because it wasn't in there earlier.
Now, the readily available and especially free certificates most people use are logged before issuance as you describe using the poisoned "pre-certificate" feature from RFC 6962 (in theory this will one day be replaced by a 6962bis actual pre-certificate document rather than poisoned X.509 certs) but that is not at all mandatory, it's just convenient because it requires no workflow changes. You get a certificate, it's a bit bigger (it has SCTs baked inside it) and it just works.
If you do things the hard way, you must ensure your server application software understands what SCTs are and can send them to clients as necessary. You save some bytes talking to clients that don't want the SCTs, and you get to have this just-in-time logging behaviour if you want that.
The other reason it isn't against policy to issue without logging is that some archaic systems that are subject the Web PKI rules but aren't actually talking to the public Internet, and especially to web browsers, do not have logging, it's not a thing for them. This will probably go away in the next few years, these systems age out, but they still existed when I last looked.
The "weird old systems" case sounds like something were maybe it could be required for DV-certs?
In effect there is already a requirement on CAs to internally track everything they issue, because there are numerous circumstances where the question, "What was issued with these criteria?" is pertinent and being unable to answer honestly is unacceptable. So, it doesn't make much sense to introduce logging as a policy requirement. I can't say it won't happen but I don't think it's useful. In contrast improvements to client software to actually examine SCTs in more clients and eventually to gossip about what SCTs they've seen are valuable because they improve the practical enforcement of logging.
What we see today is that most incidents are logged. A CA will issue something that shouldn't exist, and independent researchers will see that in the logs, but the CA's own tools didn't discover this, not only before logging but often afterwards.
Under existing policy the pre-certificate is a misissuance, even if the "real" certificate never existed we can't prove that, so the CA violated policy, but it's a problem that some of them will only realise this after it's reported by somebody else rather than flagging it immediately and self-reporting the violation. So this is reassuring because it matches my assumption that the CA operators are like most of us, they are lazy and incompetent but they aren't malevolent. They can't be bothered to do it properly, they don't remember how to do it properly, but they aren't intentionally doing a bad job, which means that they can improve if given better tools and procedures to avoid trouble.
Can you explain this further? I don't understand.
Are you saying they could MITM the request that the CA makes to the website when it tries to do domain valudation? If that's the case, why is it limited to 2 years? Why couldn't they continue to do this indefinitely?
Due to domain validation reuse[3], the certificate doesn't have to be issued right away. The attacker can wait and request the certificate up to 398 days later, without having to do domain validation again.
[1] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd...
[2] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd...
[3] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd...
This sounds like more of an Amazon problem than a MarkMonitor problem to me. And it makes a good case for using other cloud providers over AWS, as with GCP this attack wouldn't have been possible. Merely creating the DNS records pointing to the cloud provider's nameservers shouldn't be enough for anyone to then claim it and start hosting.
Bob owns bob.com and sets CNAME bob.com.us-east-1.amazonaws.com. Eve creates the bucket and can now serve any static content she wants.
There's a chance a normal AWS user may not understand the above distinction, a sophisticated actor like MarkMonitor (whose business is THIS) should. Instead they had a security incident. That's it. They used the wrong AWS service. They configured massive numbers of domain DNS records incorrectly. They risked their customer's reputations.
- DNS TXT record
- HTML meta tag
- Upload a file with a randomized name to your site root
- Bob makes bob.com and sets the CNAME to be bobstaticsite.s3.aws
- Bob forgot to make a bucket called bobstaticsite.s3, and Alice, scanning the DNS records, creates it instead. Now Alice is serving data on Bob's domain
Am I missing something? How would AWS S3 stop you from creating a CNAME? Or how is this their responsibility? Don't make that CNAME entry pointing to a bucket you didn't create yet / don't own?
One final thing: S3 is not CloudFront. And CloudFront, AWS's CDN, does require domain name verification.
And shouldn't Alice be able to register Alice.com and point it at Bob's bucket if she feels like it?
It seems like the answer already exists: don't create CNAMEs haphazardly.
(But please explain it to me if I'm missing something here)
If it blocks creation, she would have to know the bucket name Bob would want to create soon. At which point she could just create the bucket instead.
> And shouldn't Alice be able to register Alice.com and point it at Bob's bucket if she feels like it?
She can only do that with this method if Bob's bucket is called "alice.com". Doesn't seem important to support for random unrelated people, if its for a dedicated setup where Bob manages a thing for Alice you could always have a way of granting that permission specifically or to opt out of the verification.
> It seems like the answer already exists: don't create CNAMEs haphazardly.
"don't make mistakes". If people keep making a mistake all the time, that's not the greatest answer. There's a reason most other providers that allow you to point a domain at them do some verification.
Since CNAME records can point to other domains, I’m not sure how AWS is supposed to police this and allow cross-referencing from other parties. Blocking based on any CNAME presence could turn into a bucket-squatting exploit pretty quickly.
A CNAME record is an “I’m pointing my domain at some other place” record. Redirect to something out of your control, S3 or otherwise, and you’re handing over control of the returned content. That doesn’t so much strike me as a mistake but more as the way that CNAME records were designed to work.
> That doesn’t so much strike me as a mistake but more as the way that CNAME records were designed to work.
"I hold something into a saw, and it gets cut. That doesn’t so much strike me as a mistake but more as the way that saws are supposed to work. Why are all these fingerless people saying our saws should have safety features?!"
We know such things happen to all kinds of people all the time. It has been part of actual security problems. We know how to fix it, because many service providers do require validation. Why not consider the validation?
What is the actual mechanism you think should be implemented, and how does that support cross-party referencing while avoiding other modes of vulnerability (e.g. denial of service)?
They (AWS) should not accept traffic on that hostname until ownership - tightly bound - is proven.
This also helps avoid related cases, such as deleting a bucket (which now frees it up) - other services like Heroku have similarly been vulnerable to this takeover hack because of a lack of strict verification: https://0xpatrik.com/subdomain-takeover-providers/
Every new bucket, service or account handling traffic for a domain should require re-verification. That verification should never persist beyond the lifetime of that resource.
Why shouldn't they ? Don't you think it can be a useful feature ?
If someone else creates the bucket on their AWS account, I am stuck. I would have to use a different hostname, or use a more complex workaround. Since creating empty buckets costs nothing, it effectively allows DoSing hostnames from an S3 hosting standpoint.
It would be one thing to allow this behavior if a domain owner explicitly wanted it, via a TXT record or something, but as it is, it's a poorly designed solution.
That is, you'd prove you control www.example.com to AWS the exact same way www.example.com proves it is www.example.com to a web browser.
This company never created the S3 buckets at all but set them in the CNAME, for domains that were purchased but never used
They can change the CNAME at any time, it’s a pretty dumb but inconsequential exploit. Potentially some purchaser didn’t use generally free and default whois privacy and could be associated with offbrand content.
Potentially someone observant can earn a bunch of ad dollars on a popular domain.
A fun exercise could be to boot up a machine on pretty much any cloud provider and set up a simple webserver to respond to all Host-headers. Heck, I'm pretty sure you could make nginx only respond requests where both the Host-header and A/AAAA-record were correct with some Lua magic. In this scenario, would you blame the cloud provider or the administrator of the domain?
(Not necessarily you-you, but you get he idea.)
You'd naturally be capturing the requested host here, ideally along with all other request headers.
And ideally flagging unknown host headers in close to real time, so you can have fun with your visitors next time they say hello :>
The name of the bucket (which ends up being a part of the bucket's URL) should be entirely outside of the user's control. Make it a random 32 character string (and make it such that old strings can't ever be recycled).
This way people can't "register" buckets for domains they don't own.
That depends on whether you are attacking the clients or the server. If an attacker obtained a domain cert or wildcard cert, while in control of the domain, then the attacks can continue.
This is a MiTM attack that allows the fake server to be accepted by valid clients (on that network), stealing their credentials and then potentially their information from the server. There are actually many attacks that can happen here, one of which is simply to record the credentials, while passing the traffic through to the server and back from the server to the clients.
Even though the true DNS server has had its DNS record's IP address changed to point towards the correct and true server, the clients are unaware of this change, as their DNS caches have been poisoned to point towards the malicious server.
PS There is a defense against this, certificate pinning, but that is not used much in practice.
Redirecting traffic is much easier than generating certificates so a valid cert held by a bad actor can be a serious vulnerability.
This is bullshit. If you can redirect traffic you can almost always create a certificate unless you can only redirect a very limited set of traffic.
Redirecting traffic is literally all you need in order to be able to use certbot to generate a new cert.
There's no real defense against this, TLS only stops dragnet attackers.
In this case the attacker could have served anything out of the S3 bucket and conducted full blown phishing (google.ar, coinbase.ca), and generated valid certificates.
It's a big deal the order of operations here and the risk MarkMonitor introduced to all their clients.
"An Amazon S3 bucket name is globally unique, and the namespace is shared by all AWS accounts. This means that after a bucket is created, the name of that bucket cannot be used by another AWS account in any AWS Region until the bucket is deleted. You should not depend on specific bucket naming conventions for availability or security verification purposes. For bucket naming guidelines, see Bucket naming rules."
... but if you want to host a static website via S3, they explicitly say you need to create a bucket with the domain name as the bucket :shrug:
I don't get this. These domains weren't configured with CNAME DNS entries like one normally does. Does this mean MarkMonitor's parked domain name server was trying to load from S3, then falling back to something else if that failed?
S3 will just pick a bucket based on the Host header in the request, and it seems like Akamai just proxied that from the client.
That's quite a big oops. I'd love to see the panicked messages on their internal slack as they figure out what they did.
I'm not clear how the list of vulnerable domains was collected in this instance - presumably he had to know which domains to create buckets for?
At least that's my understanding of his process.
It's one of many products that supports serving under custom hostnames and all such products should have domain verification.
Because of this, I agree, they should verify domain ownership to help protect their users.
You can verify your domain is going to a cloudfront you own, but it doesn't verify that the origin is a bucket you own.
If any researcher here needs data or help to do investigations like this please reach out to me chris at securitytrails.com - we're trying to hone the tools to be as useful as possible with as little effort.
What if mark monitor would put all parked domains to bob.com.mys3.com service? Me, as mys3 provider I'm at fault? I doubt.
What if mark monitor would point to an IP address they don't own? Still not their fault?
It's purely a DNS management issue. This happens with other technologies too, like mail servers. Domains (unique identifiers like S3 bucks) expire, someone registers it, spins up a mailserver, and begins recovering accounts via password resets. Who is to blame here, the application for allowing the password reset (provided the valid email/fetched the reset key), or the person who let the domain expire? I'd say the latter, and in the s3 situation, I believe it's the same.