Hacker News Reader

Comment by orev | Hacker News Reader

The bigger problem I see is that they’re clearly using production data on test systems (or using the production system for testing), including PII. This is a pretty big no-no and violates many security standards. I don’t blame the tester per se, but I do blame HBO for not having a process in place that prevents this kind of thing from happening.

View on HN

Jump to conclusion and shame. Classic.

How about, just maybe, they use a mail list management system like millions of other companies and PII wasn’t available in tests at all?

No, no, that can't be. Don't be reasonable

Except they also sent emails to people who are not subscribers such as myself and there was no unsubscribe link in the email. There was also no header, footer or any branding at all in the email. The only content in the email was the single line of text that read:

"template is used by integration tests only."

None of that sounds like a mail list management system to me. Also nowhere does the OP appear to "shame" anyone. In fact the OP very clearly states they don't blame any person but that they felt fault lies in lack of process to prevent such incidents.

That sounds exactly like what you would send to a mail list management system. Since you’ve likely never used them, think of it like a black box you feed a template that has a bunch of vars you can reference like

“Dear {firstname},

Check out our new movie: Batman Undresses.

Thanks for being a subscriber for {accountlife}.

{termsfooter} {unsubscribefooter} {alternatelanguagesfooter} “

The whole point of a template is so you send to an entire distribution list with a single API call and the mail system handles rendering templates to per user emails, setting up the unsubscribe link, tracking pixels, etc.

> Except they also sent emails to people who are not subscribers

That has exactly zero relation ship to your name being in their mail distribution system.

> That has exactly zero relation ship to your name being in their mail distribution system.

Seems it has some relation after all: If your name isn't in the system, you wouldn't receive the result of the template being applied: Somewhere a bit higher in the code than your example, up among the headers, there's a bit like "to:{emailaddress}".

I mean, what is sending mail to non-subscribers related to, if not the contact data of those non-subscribers being in the sender's mail distribution system?

No, you’re getting confused. An email list management system has far different data than the system that dispatched the fucked up template to it. Additionally, whether or not someone is an active subscriber no impact on them being in the mail system in general.

The whole point is that the piece that screwed up and pushed this template would have no PII access itself.

Beyond it being PII, it’s just how you sanely design these types of mass email/sms/push notification distribution system.

This incident doesn't necessarily indicate that they were using prod data in a test system.

I can plausibly imagine that there's some separate system that takes an identifier for some list of customers, and some template, and blasts out emails. Such a system could exist to help manage compliance issues with e.g. unsubscribe requests.

If so, and with a few "shortcuts" taken in making test environments for integration testing, I could envision a scenario where this incident happens that don't involve the test having direct access to real user data.

HBO Max sounds like a big company (I've heard of HBO, and "Max" sounds big to me). But it is possible they are using something like Mailchimp for their mailing list and don't do it in house. I'm just guessing here - a quick look at the headers would reveal this.

And with a confusing and horrendous UI such as Mailchimp's, it's quite easy to send a test email to the "live list". VERY easy indeed.

We've done it twice now. Once to about 10,000 emails and another to almost the entire list of 800,000. Luckily the template we were testing was 95% complete and not many people noticed. It just looked like the email got truncated with gibberish at the end.

They use SendGrid. From the headers, abmail.mail.hbomax.com resolves to u6146175.wl176.sendgrid.net.

Or they ran a suite of integration tests against prod as a way to verify a release was working correctly.

but... why would you do that? production is not for that one

If you don’t test against prod your CD pipeline is missing a critical step.

In some systems the cost of writing and maintaining stubbed or alternate versions of things you don’t want hit by tests in prod can be pretty overbearing. Good SLOs can serve the same purpose as well for that sort of flow

+1. The first thought I had when I saw that email was that I felt bad for the tester/dev; it's not really their fault, but they're certainly going to get at least some backlash for this. Really, it's a director/VP level issue that this kind of mistake was even possible with my email address.

This tweet does at least give some some indication that they're not (outwardly) trying to throw their intern under the bus: https://twitter.com/HBOMaxHelp/status/1405712235108917249

It was IMO still an ill-considered tweet even if made in a jokey way. While no one was really hurt by this--OMG I got an extra email!? :-)--something more along the lines of Oops, sorry for sending out that test email by mistake. We'll be putting processes in place so it doesn't happen again.

Exactly this, though anonymizing production data properly so it can be used as test data is very hard.

Generating useful test data is much better, but it's hard to represent all the edge-cases you see in Production.

Reply on news.ycombinator.com