HBO Max accidentally sent an integration email test to users

I wouldn't go so far as to say that it's never an engineer's fault, but this sort of thing usually relates more to faulty processes than people.

I'm reminded of the engineer at AWS who was writing a bash script about 5 years ago and unwittingly took down a good chunk of AWS's main east coast region, causing outages for tens of thousands of websites. I remember admiring that their response wasn't to fire the engineer, but rather to say that any system that large that allows a single non-malicious engineer to take it down must need some beefing up to make that sort of mistake impossible.

That sinking feeling when you realize you've started something bad and can't stop it always gives me a visceral feeling like the world is doing a dolly zoom* around me.

I have had two really fun bulk email screwups:

The first one started when we hit the the send button for a mass email campaign driving traffic to our newly launched website redesign. We immediately realized that the email marketing software had put a unique query parameter in every link that was resulting in all requests to our cache going to origin, which instantly smoked the little vm hosting the site, sending thousands of clicks to the now famous 503 Guru Meditation page. With the infrastructure folks offline in another timezone, it was the perfect environment to learn Varnish Configuration Language on the fly with the whole marketing team hanging over me with looks of horror on their faces!

The second one involved coming to work, sitting down with my coffee and noticing that our email sending process had crashed overnight. Given the rate they could be sent sequentially I realized we'd have a big backlog of tasks, so I wrote a quick shell script to split the tasks into separate lists and parallelize them across the number of cpus on our (big, colocated, dedicated, hosting many important apps and websites) server. As soon as I ran it I sent it to the background and opened up `top`, only to see thousands and thousands of forks of my process filling the list. By the time I realized that I'd flipped the numbers and split 4 email tasks into each of 50,000 processes instead of 50,000 tasks into 4 processes, the server locked up and my SSH session disconnected. Cue several panicked minutes of our apps being offline while I scrambled for the restart button in the remote management console. Somehow there were no lasting effects, and every service started up on its own when the box came back online, even though it hadn't been restarted in several years.

* https://filmschoolrejects.com/wp-content/uploads/2021/01/Jaw...

Every self hoster's nightmare holy cow.

GUI clients might be destroyed, but wasn't writing an IMAP script to bulk delete all emails an option?

I came out as trans and changed my name last year, and with the name change I set up a new email alias for work. Then I set up automation to send out a gentle reminder email about the change for people who emailed my old alias. It worked fantastically for a few months… right up until the point that (due to a series of individually innocent events) the automation ended up running across the entirety of my 7 years worth of inbox. Everyone who had mailed me over the 7 years prior to the name change started getting the reminder email. One reminder for each email they had sent me. The worst part is that due to a bug in the email automation stuff, emails sent by the automation weren’t preserved in my sent box. So I don’t even know how many people I spammed. If I had to guess, I sent dozens of emails to the CEO and other execs, hundreds to my director, and thousands to people who worked closely with me over the years.

I learned a valuable lesson that day.

On the bright side, if you can accidentally send an automated email to that many people, then sending another email to them to apologise is unlikely to be a manual effort either.

I hope not. It sounds like the test database was not being anonymized but sometimes things like this can be as simple as not selecting a debug build in Visual Studio, either of which is an organizational issue and not an individual one so he shouldn't be punished.

This is especially true if you have corporate buzzwords like "taking ownership and responsibility". No one will take responsibility if there are punishments for owning up to and admitting mistakes. Odds are they feel pretty bad about it already.

While we're sharing personal anecdotes. Parents get very upset when they incorrectly receive truancy reports because you forgot to check the IsDeceased flag...

Now I know better and if somehow they are reading here, I would advise the person to just chill and don't take any shit than necessary. If you were not one of the few people with root access but somehow still had the capabilities for mass emailing in prod, that is not your problem, it is an organizational problem. For an operation at the size of HBO, anything prod has to be behind sufficient failsafes and a peer reviewed process (except maybe for a very rare "break glass" emergency).

Hope there will be a good, rational postmortem that can cool headedly identify the root causes and create action items for the actual stakeholders. If your shop is worth its salt, there wouldn't be performance evaluation consequences for you. If there is, no worries either, it is time to look for a better place.

My script was supposed to try to grab batches of 1000 customers and keep looping until it ran out of customers (signaled by having retrieved less than 1000 customers in my last request for the next batch of 1000 customers).

My script was missing the offset part of the query, so after we hit 1000 users, it just kept looping, sending the same email over and over to our first 1000 users.

I felt so bad that day. From then on, sending out emails was this whole huge process that involved queuing them all and then having like 6 people review to make sure we didn't mess it up.

We've all done it. If you haven't broke production you haven't broke production yet.

https://twitter.com/HBOMaxHelp/status/1405712235108917249

I learned this the hard way in my last role, I worked for a small company that wanted to do custom email marketing. I was pretty gung-ho about it, I thought I'd just set up a script to loop through contacts and use mailgun to send the email from a custom domain. As we used the tool we saw a steady drop-off in click-throughs to the site, the majority of our messages were getting caught up in spam filters.

Turns out there's a whole science to email marketing, how emails should be structured and formatted etc. A lot of times the criteria for spam is how often the domain was flagged for spam in the past, the length of the email, the contents etc.

Ended up relying on this tool quite a bit: https://www.mailgun.com/deliverability/email-spam-checker/

It was a placeholder until the software dev team responsible for the server could deploy software onto it (they asked it be setup with something in the index.html so they could confirm it was running, they didn't care what)

The networking team in advance of that wired it up to some load balancers.

They took an IP which had formerly been a decommissioned cluster of webservers serving the main website for the entire company (an internet retailer named after a river in some other country, you've probably never heard of them).

The DNS loadbalancers found the old IP (it had never been deleted from their configs which was the root cause) was now live and it was REALLY fast, so they shunted most of the traffic over to it.

Created a sev1 outage for "users reporting 'foo' on the website"

I'm happy I kept it professional that day.

I'd semi-automated the generation of emails to parents about how their kids were doing at summer camp. It was basically just a script that asked various questions and pulled various data about the camper from the database, then used that to generate a letter that could be used as a starting point and edited to be more personal (or, if in a rush, just sent as-is).

Long story short, due to user error, a value of 0 was set for a camper's behavior/politeness/helpfulness rating, which resulted in a joke sentence that I'd written as an Easter egg getting slipped into that particular report. Cue egg on my face when a parent calls in, baffled about the otherwise normal report containing a casual aside about how terrible their nice little girl was.

After that experience, I always assume that every string in the code will inevitably be seen by a real user/customer.

    This is a test.
    This is only a test.
    Had this been a real emergency, we would have fled
    and you would not have been informed.

A previous company I worked in once managed to send the following in patch notes:

"So sit back, smoke a spliff and stop worrying you nappy wearing fucktards"

... so, yeah, good advice.

I learned my lesson from him and now I keep these things very clean, just in case.

There should have been a label on the computer near the keyboard jack written by the people who knew about the problem.

If you didnt make a label afterwards yourself, the next keyboard-based outage would be your fault :)

The server restarted and just continued happily doing its job.

It looks a lot like this https://xkcd.com/908/

I have also seen instances of similar things, but usually there is a large sign saying "do not unplug/turn off" and they try to make it as unobtrusive as possible.

The problem here is that it is you were the newbie. It should have been done by a higher up who could then properly chastise whoever was responsible of that mess.

The first you could apologize in many ways including financial, the second you really can only apologize by doing it again.

... (but I'll say the rest anyway). The system was using dozens of phone lines to call me and leaving multiple voice messages simultaneously. I considered myself reminded.

In AOL one prank was to flood someone with emails because it was difficult to remove them, IIRC and needed to be done almost one by one.

I can plausibly imagine that there's some separate system that takes an identifier for some list of customers, and some template, and blasts out emails. Such a system could exist to help manage compliance issues with e.g. unsubscribe requests.

If so, and with a few "shortcuts" taken in making test environments for integration testing, I could envision a scenario where this incident happens that don't involve the test having direct access to real user data.

Generating useful test data is much better, but it's hard to represent all the edge-cases you see in Production.

How about, just maybe, they use a mail list management system like millions of other companies and PII wasn’t available in tests at all?

There's a pretty good chance the person who made such a mistake is less likely to repeat it or other mistakes than someone who hasn't yet had the experience. So at best you're just sending a more valuable employee to one of your competitors.

There's some gray area when it comes to making a poor judgement call to skip a well defined procedure or be lazy.. and some cases this doesn't apply to like intentional malice or a continued series of similar obviously avoidable mistakes.

On a related note im often frustrated when people play a blame game (even if one might be somewhat called for, in a way) instead of considering "human factors" and how to systematically avoid people being able to make such mistakes. Or refuse to accept a human factors style explanation and just say the person "should have known better" etc. The best high profile example was that hawaii missile alert that was supposed to be a test having an incredibly poor UI. But there are countless examples everywhere all the time. I often try to think about this a lot in my work.

My PS4 HBOmax app won't run, won't uninstall, and won't update. The only way out I can see is reinstalling the system software. It's possible it's a PS4 issue, but none of my other software has this problem. If the PS4 was the only way I could consume my subscription, I'd unsubscribe.

So, I'm not really impressed with their technological kitchen.

How do you find out whether a user reports you as spam? Wouldn’t that require some sort of feedback from Google, Microsoft, etc?

I hope that their apparently compassionate public stance translates into a healthy internal response that doesn't railroad the poor intern.

Did you know only a handful of US states have laws about how you can use or handle social security numbers? If the business isn't in one of those states, expect the intern to have a big list of SSNs on their laptop. Even when there is a regulation, sometimes it's intentionally violated on a regular basis, either as part of a cost/benefit analysis, or some loophole that means nobody will personally be held responsible.

"""

If you are a California resident, California Civil Code Section 1798.83 permits you to request information about our practices related to the disclosure of your personal information by certain members of the WarnerMedia family of companies to certain third parties for their direct marketing purposes. You may be able to opt out of our sharing of your personal information with unaffiliated third parties for the third parties’ direct marketing purposes in certain circumstances. Please send your request (along with your full name, email address, postal address, and the subject line labeled “Your California Privacy Rights”) by email at WMPrivacy@warnermediagroup.com.

"""

https://www.hbomax.com/privacy

I'm actually hard pressed to even imagine the person that is cancelling their HBO max account because this single email was just too much for them.