Hacker News Reader

Comment by mcherm | Hacker News Reader

I just want to make clear that for me, write ups containing this level of detail analyzing the error that occurred give me greater confidence in the company rather than less confidence. Greater even than if I had never heard of the original bug. There are many bugs I have never heard about, but any company that uses this level of care and openness is likely to be very reliable.

View on HN

I have to say the presentation is exceptional, including very good diagrams of the problem.

But there's a thought in the back of my head wondering what kind of stuff hit the fan, and wondering who had to create the presentation. Was it the person who checked in the broken code, was it the bug fixer or his manager, or was it a pm that got stuck having to write up what happened?

My guess is that someone close to the remediation wrote it (or it was written by the eng team) and it went through 5 levels of editing before publication.

I wonder about the email thread. how many recipients on the TO, how many on the CC (how many on BCC?)

> how many on BCC?

I guess the only way we will know that is if there’s ever a vulnerability in the session handling code of the mail server that they use also xD

I agree with you in general, but in this particular case they knowingly deployed thread-unsafe code in production while increasing the threaded concurrency. That should have broken internal development rules.

Now imagine what happens behind the scenes at other service providers. I’d bet the ones that have policies that bury the details of incidents, on average, have worse problems.

It sounds to me like they created new internal development rules as a result, which is a smart response to the issue.

Yes. Writing code with bugs breaks internal development rules. The issue wasn't thread-unsafe code. It was confusion about object lifecycle. An env object was assumed to no longer be used by the web framework after a request terminated, while in reality it was recycled for future requests as a performance optimization.

Agreed. However, I smirked a bit when I read your comment, since this is ultimately coming from Microsoft. Let's hope some of this mentality seeps into other Microsoft projects

GitHub is owned by Microsoft but as far as I know it operates pretty independently. Let’s hope it stays that way.

At this point we're all just waiting for that last E.

Or, pro-actively moving to GitLab - either self-hosted or paid and using GH as a simple mirror for discovery.

I don't understand this reference, can you elucidate

Most likely a reference to Embrace, Extend, Extinguish. A strategy Microsoft employed in the past.

Ah thanks, yeah. Familiar with the strategy, but didn't get the connection - been a long day over here

We should Eulogize

https://en.m.wikipedia.org/wiki/Embrace,_extend,_and_extingu...

elucidate, extend, extinguish

What would be the benefit for Microsoft in that case? - They are transforming themselves from a platform company based around windows, it a platform company based around Azure. GitHub is a relevant part there.

This doesn't mean they are only doing good™ with GitHub, but destroying it and open source vendors would be counted to their interests.

This is such a n00b mistake though, the "greater confidence" is quickly canceled out. I end up having less confidence in this company.

Every bug is a "n00b mistake" when you are looking right at it. There is not a single large project that is not filled with or at least has had trivial bugs with serious repercussions.

> Every bug is a "n00b mistake" when you are looking right at it.

Except the ones that are just the universe conspiring against you, like bit flipping in the SM-1800 because of heavily irradiated cattle being transported on a nearby train line.

If you’re a noob then yes every mistake is a noob mistake

I think it's more a statement of the quality and clarity of the postmortem that this problem appear so obvious in hindsight.

Reply on news.ycombinator.com