Hacker News Reader

Comment by JumpCrisscross | Hacker News Reader

> you bear some responsibility for the resulting mess

But is it a compromise of the app? I'm failing to see the nexus between Houseparty, the app, and these subdomains. How does someone using the app end up on the subdomain?

View on HN

This is speculation, but I think reasonable:

- Houseparty spins virtual servers up and down quickly to respond to changing demand.

- They have a DNS TTL that's longer than the draining time for those VMs. Like, maybe they it takes them five minutes to kill a VM but DNS records are cached for an hour.

- The attacker takes advantage of that window to spin up a bunch of tiny VMs and see if any of them are allocated one of the IPs that Houseparty recently abandoned. If so, then they fire up Nginx and serve poisoned PDFs to anyone who connects to it.

- An end-user's app connects to "ephemeral-vm-837.houseparty.whatever" and gets the cached value that their ISP is still serving, because Epic configured their DNS to tell the ISPs to cache it for an hour, except now that IP is served by the attacker instead of by Epic.

Mitigating this could be as blunt as setting DNS TTLs to 5 minutes, then putting "sleep 300" at the start of each server's shutdown script. When you decide to kill a server, first kill its DNS record so that nothing refers to it anymore, then start the waiting period before actually removing it from rotation.

> If so, then they fire up Nginx and serve poisoned PDFs to anyone who connects to it.

Even if the attacker redirects the traffic, it still needs to serve the HTTPS/SSL certificates. How will the attacker do that?

Maybe it does? It could be serving plaintext HTTP ("why would we encrypt our logo.png file?"), or the client library could be configured to accept any valid SSL cert because they don't want to deploy good certs to every ephemeral host they spin up.

There are a lot of ways this could go wrong.

Agreed. In my mind, if the server is using plain HTTP, and the attacker has control over parts of the network, then plain-old DNS is not the only attack model. The attacker does not need to hack DNS, they can just manipulate the content on the wire.

I believe LetsEncrypt will issue you a certificate for that subdomain if you can control the server responding at that subdomain.

EDIT:

Specifically: https://letsencrypt.org/docs/challenge-types/

It won't give you a wildcard certificate, but you don't need one for the type of attack we're talking about.

Well, hopefully they use certificate pinning. And if not, they should do certificate transparency like chromes does.

You can bring your own IP range or pay for an elastic IP. Isn't this how folks do this (I do!).

You get a bunch of elastic IPs, you point your domains to those, you point those to your instances.

That's secure, no one can use those IPs until you release them. Is this rocket science? On AWS I don't think you really get charged for this even (or very little) from what I can tell if you have them all pointing to your instances.

Yup. Used elastic IP addresses are mostly free[1]. Unused elastic IP addresses are $0.005/hr[2]. Although watch the remapping fee... $0.1 per remapping past the first 100. That could add up.

[1] https://aws.amazon.com/premiumsupport/knowledge-center/elast...

[2] https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Add...

That would go a long way, but note that AWS limits you to 5 IPs per account per region[0] by default, and charge[1] $0.005 per Elastic IP per hour for EIPs that aren't attached to a running instance, and $0.10 to transfer an EIP from one instance to another. So, if you want to keep 1,000 EIPs around to map to ephemeral VMs, and you're not actively using them, you'll be paying $3,600 per month for those alone.

One could argue that this is pocket change for a large customer to do the right them, but from a PM's perspective that $40K per year they could be spending towards another engineer's salary.

[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-... [1] https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Add...

> Mitigating this could be as blunt as setting DNS TTLs to 5 minutes, then putting "sleep 300" at the start of each server's shutdown script. When you decide to kill a server, first kill its DNS record so that nothing refers to it anymore, then start the waiting period before actually removing it from rotation.

If I had a dollar for everytime traffic stopped to an IP after the DNS ttl expired, I wouldn't have any dollars. After taking IPs out of DNS for www.popular site, I would continue to see traffic for weeks, and the TTL was set for somewhere like 5 minutes, an hour tops. Edit: also, when we moved our authoritative DNS, the old provider kept seeing queries for at least 6 weeks.

If the only thing preventing you from sending private data or receiving trusted data is that it came from an IP you got from DNS, it's not secure.

I think what matters is whether the certificate authorities honour TTL correctly when they do domain-control validation.

If they do, then the procedure above (and doing everything over https) should be enough, I think.

Oh, buddy, you know that's right. There are plenty of IPSs like, "5 minutes, lol! Two weeks it is!"

What I proposed was a blunt force tool to try to help mitigate the problem, but you're 100% correct that it's not sufficient. Key pinning would go a lot further towards fixing it. What I suggest could be rolled out pretty quickly and without risk, though, so IMHO it's worth adding to the to-do list.

This is what TLS certificates solve. They make sure the other end of the IP connection has the name you expect it to have.

If they've hijacked a subdomain, they can submit it to LetsEncrypt and get a cert. LetsEncrypt checks that you control a domain by asking you to serve some data from it. The domain is already hijacked so from LetsEncrypt's perspective it's yours.

Who cares if it is a compromise of the app or of their methodology for hosting content? I think it's still their responsibility to make sure their subdomains are not hijacked. What would happen if google.com let one of their IPs slip like that?

It’s bizarre to me people are making such a strong distinction between backend infrastructure and client app. Almost feels like some pre-defense mechanism?

You target people using the app. If you had control of verify.paypal.com you'd target people using paypal apps. Even though it's not a compromise of the app argueably it's a concept of the app platform. Users's don't make these distinctions. They install the app, they get a message on the app to go to the apps domain to verify their info (email / 2 factor authentication).

They are NOT then checking that the subdomain was released and got a new IP recently (or didn't get an IP).

These PR people love to split these hairs. Our app is secure - but we leaked 100% of your data out of our back-end servers when they were hacked for 3 months - is another one that's common.

Arguably, yes. At a minimum, poor security measures.

"Certificate Pinning" and/or HSTS both help, and there are other measures.

Reply on news.ycombinator.com