The mystery of 'hacked' Houseparty users may have been solved | Hacker News Reader

The mystery of 'hacked' Houseparty users may have been solved

medium.com

171 pointscududa48 comments

We changed the URL from https://gizmodo.com/the-mystery-of-hacked-houseparty-users-m... to the original source it points to. It also points to https://www.theregister.co.uk/2020/05/20/houseparty_subdomai....

Switching to the original source seems fairest, but you might want to read one of those other articles too because they give more of the backstory.

I've complained about this in the past (and was attacked for it). I'll do it again.

If you create a subdomains under your trusted domain, and point those domains to IP address controlled by scammers - you bear some responsibility for the resulting mess.

The Epic claims are pure PR spin. Epic claimed

"these rumors amounted to a paid commercial smear to harm Houseparty, announcing on their Twitter account, that a million-dollar bounty was being offered to “the first individual to provide proof of such a campaign.”"

An individual offered clear proof that Epic was pointing houseparty domains to scammers who were using that to steal folks info, the exact situation users were complaining about. These are domains UNDER EPICS TOTAL CONTROL being pointed to scammers.

They now claim they still found no evidence of a campaign against houseparty users by a hacker group - and this was all a "paid smear campaign".

Who believes these PR people -seriously?

If Amazon takes payments.amazon.com and point it to Scammer IP XXX, how does that NOT create a vulnerability?

> you bear some responsibility for the resulting mess

But is it a compromise of the app? I'm failing to see the nexus between Houseparty, the app, and these subdomains. How does someone using the app end up on the subdomain?

Arguably, yes. At a minimum, poor security measures.

"Certificate Pinning" and/or HSTS both help, and there are other measures.

This is speculation, but I think reasonable:

- Houseparty spins virtual servers up and down quickly to respond to changing demand.

- They have a DNS TTL that's longer than the draining time for those VMs. Like, maybe they it takes them five minutes to kill a VM but DNS records are cached for an hour.

- The attacker takes advantage of that window to spin up a bunch of tiny VMs and see if any of them are allocated one of the IPs that Houseparty recently abandoned. If so, then they fire up Nginx and serve poisoned PDFs to anyone who connects to it.

- An end-user's app connects to "ephemeral-vm-837.houseparty.whatever" and gets the cached value that their ISP is still serving, because Epic configured their DNS to tell the ISPs to cache it for an hour, except now that IP is served by the attacker instead of by Epic.

Mitigating this could be as blunt as setting DNS TTLs to 5 minutes, then putting "sleep 300" at the start of each server's shutdown script. When you decide to kill a server, first kill its DNS record so that nothing refers to it anymore, then start the waiting period before actually removing it from rotation.

> If so, then they fire up Nginx and serve poisoned PDFs to anyone who connects to it.

Even if the attacker redirects the traffic, it still needs to serve the HTTPS/SSL certificates. How will the attacker do that?

Maybe it does? It could be serving plaintext HTTP ("why would we encrypt our logo.png file?"), or the client library could be configured to accept any valid SSL cert because they don't want to deploy good certs to every ephemeral host they spin up.

There are a lot of ways this could go wrong.

Agreed. In my mind, if the server is using plain HTTP, and the attacker has control over parts of the network, then plain-old DNS is not the only attack model. The attacker does not need to hack DNS, they can just manipulate the content on the wire.

I believe LetsEncrypt will issue you a certificate for that subdomain if you can control the server responding at that subdomain.

EDIT:

Specifically: https://letsencrypt.org/docs/challenge-types/

It won't give you a wildcard certificate, but you don't need one for the type of attack we're talking about.

Well, hopefully they use certificate pinning. And if not, they should do certificate transparency like chromes does.

You can bring your own IP range or pay for an elastic IP. Isn't this how folks do this (I do!).

You get a bunch of elastic IPs, you point your domains to those, you point those to your instances.

That's secure, no one can use those IPs until you release them. Is this rocket science? On AWS I don't think you really get charged for this even (or very little) from what I can tell if you have them all pointing to your instances.

Yup. Used elastic IP addresses are mostly free[1]. Unused elastic IP addresses are $0.005/hr[2]. Although watch the remapping fee... $0.1 per remapping past the first 100. That could add up.

[1] https://aws.amazon.com/premiumsupport/knowledge-center/elast...

[2] https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Add...

That would go a long way, but note that AWS limits you to 5 IPs per account per region[0] by default, and charge[1] $0.005 per Elastic IP per hour for EIPs that aren't attached to a running instance, and $0.10 to transfer an EIP from one instance to another. So, if you want to keep 1,000 EIPs around to map to ephemeral VMs, and you're not actively using them, you'll be paying $3,600 per month for those alone.

One could argue that this is pocket change for a large customer to do the right them, but from a PM's perspective that $40K per year they could be spending towards another engineer's salary.

[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-... [1] https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Add...

> Mitigating this could be as blunt as setting DNS TTLs to 5 minutes, then putting "sleep 300" at the start of each server's shutdown script. When you decide to kill a server, first kill its DNS record so that nothing refers to it anymore, then start the waiting period before actually removing it from rotation.

If I had a dollar for everytime traffic stopped to an IP after the DNS ttl expired, I wouldn't have any dollars. After taking IPs out of DNS for www.popular site, I would continue to see traffic for weeks, and the TTL was set for somewhere like 5 minutes, an hour tops. Edit: also, when we moved our authoritative DNS, the old provider kept seeing queries for at least 6 weeks.

If the only thing preventing you from sending private data or receiving trusted data is that it came from an IP you got from DNS, it's not secure.

I think what matters is whether the certificate authorities honour TTL correctly when they do domain-control validation.

If they do, then the procedure above (and doing everything over https) should be enough, I think.

Oh, buddy, you know that's right. There are plenty of IPSs like, "5 minutes, lol! Two weeks it is!"

What I proposed was a blunt force tool to try to help mitigate the problem, but you're 100% correct that it's not sufficient. Key pinning would go a lot further towards fixing it. What I suggest could be rolled out pretty quickly and without risk, though, so IMHO it's worth adding to the to-do list.

This is what TLS certificates solve. They make sure the other end of the IP connection has the name you expect it to have.

If they've hijacked a subdomain, they can submit it to LetsEncrypt and get a cert. LetsEncrypt checks that you control a domain by asking you to serve some data from it. The domain is already hijacked so from LetsEncrypt's perspective it's yours.

Who cares if it is a compromise of the app or of their methodology for hosting content? I think it's still their responsibility to make sure their subdomains are not hijacked. What would happen if google.com let one of their IPs slip like that?

It’s bizarre to me people are making such a strong distinction between backend infrastructure and client app. Almost feels like some pre-defense mechanism?

You target people using the app. If you had control of verify.paypal.com you'd target people using paypal apps. Even though it's not a compromise of the app argueably it's a concept of the app platform. Users's don't make these distinctions. They install the app, they get a message on the app to go to the apps domain to verify their info (email / 2 factor authentication).

They are NOT then checking that the subdomain was released and got a new IP recently (or didn't get an IP).

These PR people love to split these hairs. Our app is secure - but we leaked 100% of your data out of our back-end servers when they were hacked for 3 months - is another one that's common.

I believe GitHub had a related vulnerability years ago before GitHub Pages had their own domain.

howdy - the hijackers didn't purchase anything. the subdomains that were compromised were * .thehousepartyapp.com // that domain had full privileges on houseparty.com in their CSP policy due to it being written as https://*.thehousepartyapp.com // whenever a user logged into HouseParty.com or did a password reset or anything with the authentication systems, they were making requests to endpoints on thehousepartyapp.com -- the 2-domain architecture was extremely questionable from the get-go, but became outright dangerous when they lost control of the subdomains on their authentication domain. cheers~

You're the Zach Edwards behind the findings I assume? Helpful to confirm in case others miss the username implication!
(Nice work!)

thank you! yup that's me

Epic claimed that this was basically a commerical smear campaign, and offered a million dollars if anyone would prove it was true.
Someone proved that it wasn't a commercial smear campaign, it was merely hackers hacking for the lulz and to get some free pizza and porn.
But Epic only promised $1m if you proved it was a commercial smear campaign, not $1m if you merely figured out the origins of it.

Such a win-win situation for Epic then, they weren't interested in the truth, they were interested in their truth, if that got confirmed by third-party they were right all along and could say so, if not, well, they aren't out of $1m then.

Is that bad?

Acting like a victim instead of cleaning up their own mess is that special mix of defiant, disingenuous incompetence that is still bad, even though it has become the standard way of responding to issues.

But they already claimed they believed it was a commercial operation. So they're going to take a reputational hit at least by some for being conspiracy theories.

Probably a much more common problem - definitely need to ensure that all resources are linked together through the DNS provider so that when you destroy the VM you also remove the associated DNS record if it is the last available resource.

Wow this is a really detailed overview. I discovered something similar myself late last year. tsn.ca (large canadian sports broadcaster) had a subdomain taken over and they used the SEO spam strategy. The subdomain was a azure address, so I figured TSN deleted the azure app but forgot to remove the DNS redirect to the internal azure URL. I never figured out what the attackers were using this for and when I checked recently the subdomain had been recovered. (checkin.tsn.ca for anyone curious, it will still appear in google)

After skimming the story, I'm still unclear about a few details:
(1) In whose judgment must the argument count as "proof"? Epic's, or a court of law's, or someone else's?
(2) With $1M at stake, it could make sense to sue Epic for the bounty. Is there some legally valid contract that lets Epic avoid that?

Just deleted the app. Never again. Gotta vote with your feet!

> I was quickly shocked with their lack of protections for users and tweeted a brief detail about a lack of “Content Security Policy” (CSP) request headers for the login form that would prevent MiiM phishing attacks — their login form could be embedded on a 3rd party website without it breaking:
Huh, what? CSP headers are awesome, but afaik not against MitM attact, nor against embedding the page in 3rd party webpage... Or am I missing something?

Wow, this author needs to be more concise. That article keeps going and going, with gigantic fills-my-1080p-monitor screenshots that seem to be low on content, that I lost the thread of what he was trying to convey.

As far as I can tell, they had stale dns records pointing at IPs that were eventually used to run scammy web servers.
I couldn't find any connection to a breach.
The author does correctly point out that the CSP headers were overly permissive. If the new owners of the IPs knew what they had, they could definitely have used that to find other flaws and exploit them.
I think this situation would need to be combined with XSS on an active Houseparty domain to be effective. The weak CSP would make XSS or CSRF possible.
Edit: if authentication tokens were stored in cookies that span subdomains, it's also possible sessions could be leaked that way.

YMMV. Maybe a little editing but I like having long-form and depth.

I'm just seeing big blank spots where I suppose there are supposed to be Twitter embeds in this article? Anyone else?

Yet another click-bait headline - on the verge of outright misrepresentation.
The tweet in question (https://twitter.com/houseparty/status/1244827034406121472) clearly offers the bounty for "the first individual to provide proof [that the recent hacking rumors were spread by a paid commercial smear campaign to harm Houseparty]". The article goes to great lengths to explain this is exactly what didn't happen and so no bounty makes sense.
Edit: I just noticed the headline of the article is actually more correct. Maybe someone can fix the title on this post. As I write this comment, the title on the post is "Epic refuses to pay out $1M bug bounty promised for discovering exploit"

If you email the mods using the footer Contact link, they'll consider the change you propose.

Yes, that was egregious. Submitters: rewriting titles like that is against the site guidelines and will eventually cause you to lose submission rights on HN.
The guideline says to rewrite titles only to make them less baity and misleading—not more.
https://news.ycombinator.com/newsguidelines.html

Title is clickbait and doesn’t match the article.

Fixed now. https://news.ycombinator.com/item?id=23251196

HN headline is extremely editorialized.
EDIT>>Misunderstood the exploit.

How does failure to secure your backend absolve the product

Fixed now. https://news.ycombinator.com/item?id=23251196

Oh, this is the same Epic that is attempting to become the dominant game store by bribing developers with Chinese money made from free to play mobile games.
Do give them your credit card info...

Reply on news.ycombinator.com