Update on Christmas Issues | Hacker News Reader

Update on Christmas Issues

store.steampowered.com

90 pointsdom9627 comments

   "There are only two hard things in Computer Science: 
    cache invalidation and naming things."

   -- Phil Karlton

You forgot off-by-one errors

I believe the quote is "There are 2 hard problems in computer science, naming things, cache invalidation, and off by 1 errors."

This is a famous quotation but in my experience it is not true. I have implemented web caching a few times and while some RPC-style forms/APIs have been difficult I've never put a cache in and had it go seriously wrong. Naming things is sometimes tricky but never profoundly difficult.

The hard problems are probably things like doing something concurrently or maybe implementing the right feature...

It's possible when you don't realise that something will be cached (for whatever reason - incorrect verb, missing etag or cache-control headers, some cookie confusion), and suddenly everyone starts seeing one person's experience (hopefully only a slightly tweaked page, but at worst their account screen).

Turning the caching off is easy, but it can be really difficult to track down the cause after that because you tend to need a specifically-timed race condition, and even when you're sure you've fixed it everyone is really nervous to turn the cache back on again...

"... it is not true. I have implemented web caching ..."

As the quote says, it is not the implementing which is hard, it is the invalidating part that is little bit tricky.

If you try to invalidate the cache in a strict manner, you will not enjoy the benefits of the caching much, since it will invalidate most of the time. If you have more relaxed implementation of it, there is a risk of this exact problem happening.

It sounds like some people forgot a `Vary: Cookie` header while they were all running around with their hair on fire because of the DDoS.

I, and many other ops-ish people I know, cringed at the obvious caching error that seemed like the most likely culprit.

The "in the middle of a DDoS" makes it make a lot more sense, and while this is still a cringeworthy situation, the cringing at least on my part is now more sympathetic than horrified.

The hardest is gathering requirements...

The third is refraining from posting this quote when one of those two issues comes up.

This is a pretty solid public postmortem. Props to Valve. Usually when something bad happens in the gaming realm, it just gets swept under the rug.

Valve is (was?) known for being silent too often - at least concerning Dota 2 (the diretide event 'scandal'). Maybe they're trying to correct that, after so many complaints about their poor communication, which is nice.

I think this was big enough that they had to comment. Publicly leaked personal information might be enough to start a lawsuit (IANAL). But I agree, props to them for going against their SOP, and actually communicating.

I really hope this catalyzes Valve to create a proper customer support team, one that can publicly address these issues in a timely manner as well as handle day-to-day customer support problems. It's ludicrous that a company as big as Valve has got away with it for as long as they have, and that's due in large part to an extremely devout fanbase that's only now starting to get fed up with them.

I don't think the Valve fanbase is even close to being fed up with them, unfortunately.

Unfortunately, there's just no competitor big enough to force them to care

gog.com is becoming really popular, and I actually had a really good experience dealing with EA Origin's support earlier this week. Sure they're not as big as Steam yet, but they're getting big enough to be competitive.

Nice to see the update and I'm very interested in how many people they determine deserve compensation, and why them (if there is any distinction between severely affected and no-big-deal effects). The error seems "basic"...in the sense that everyone who knows devops knows how to prevent it, but according to their account, the circumstances were extreme (plus, it was the holidays) and so mistakes were made. Just like how only a seemingly moron of a surgeon could operate on the wrong limb of the wrong patient...but such catastrophic mistakes often happen when the ER is in a state of chaos.

>Valve is currently working with our web caching partner to identify users whose information was served to other users, and will be contacting those affected once they have been identified.

How will they be able to do that?

Analyzing log files, I presume.

They're glossing over over this a bit. Many sites use the last 4 digits of your cc number as validation so the last 2 digits + email + phone suffix setup is valuable information and is definitely an addition to any cc database for liveliness of credit cards or accounts to go after on steam.

Do people get fired for making these kind of mistakes?

Probably not, unless it's one of a series of similar issues. Hiring and training people is expensive, and everyone screws up sometimes.

Firing people for a single mistake is for fast food restaurants, not for billion dollar tech companies.

They didn't make a mistake; they hopefully learned a huge lesson. It's hard finding people already having learned this lesson, so you shouldn't fire over this incident alone.

While an expected response, it's hard to forgive Valve for taking 5 days for any official acknowledgement of the issue (not even a Tweet on the Steam Support Twitter accounts, which you would expect to be the primary source of information), especially since personal information leaked.

To quote a NeoGAF moderator:

> To be honest, I think people would have been willing to give them 5 days (irrespective of what the professionally appropriate timeframe was) if their initial response wasn't so tone-deaf.

Valve did make an official statement to the press on the day of[0].

[0] http://www.pcgamer.com/warning-steam-is-revealing-private-ac...

Call it petty, but hearing their official statement secondhand makes it feel disingenuous to me. They have more appropriate channels in place for that sort of thing, including email, their two official Twitter accounts, and their very own web site, which would communicate directly to the customers affected.

Reply on news.ycombinator.com