A human error caused some sensitive server... | Hacker News Reader

A human error caused some sensitive server...

staff.tumblr.com

49 pointsGVRV18 comments

Since all that really can be deferred from the dumped code is that someone made a typo in a globally included config file (whether it was outside of webroot or not can't really be determined). The obvious issue is that obviously there was no process for testing code prior to deployment. This could have been caught using a standard Dev/Stage/Production environment setup, or even a quick test prior to deployment to Production. Other possible ways to catch this is with proper Unit Testing (which would have broken when not a single service would work due to no config params coming through), or a proper Continuous Integration environment to handle all that.

I'm really beginning to wonder what the hell is going on at Tumblr. Constant downtime to where their down message is the new fail whale, cases of severe downtime with poor communication, and now this.

Hopefully the shape up. I'd have serious concerns even running a personal blog on there.

Aren't all bugs effectively a human error at some level? In this case, the human wrote bad PHP.

I hope Tumblr will keep us more in the loop on this than when they were down for 24+ hours.

no?

Bugs can arise when a system or code base is being used in a manner that was not originally envisioned by its author.

That's not a bug, that's something more akin to failure due to scope creep.

I fail to see how that isn't also a human error?

Yes, but it's a different and more serious kind of human error.

"Someone made a typo" is one kind of human error. It's common and easily corrected. "We are running our servers such that someone can ruin everything by making a typo, and we won't catch it until it's too late" is a different, more serious and less easily-corrected error.

But yes, all bugs are human error.

Well, excepting cosmic ray bit-flipping and the like.

And sometimes it's just a bug. http://www.history.navy.mil/photos/images/h96000/h96566k.jpg

No, all bugs are not human error.

Some bugs are caused because the programmer cannot see into the future and determine that something they programmed will be used in a manner that was not intended (using a Hammer function instead of the new Screwdriver function).

So the bug is in the code that the programmer wrote years ago. It is not suitable for today and WOULD crash and create a bug if used that way.

But is there an error somewhere? No, the programmer just cannot see into the future.

Whoever writes the code that makes it possible for the Hammer to be used when Screwdriver is called for during execution is introducing a bug into the codebase... It's a bug caused by a human, not necessarily the same one that created the Hammer.

Ah, but that is human error - the error of the person who used a program in a manner for which it was not intended! They erred in not understanding that the program was written years ago and not intended for that use. Of course, then it is not the error of the human who wrote the program.

This, however, is the reason why the phrase "human error" is mainly applied to simple, 'random' mistakes such as typos, fat-fingering a button, or misfiling a paper - errors that anyone can make, simply by being a forgetful human - rather than the literal meaning of "any error committed by a human". Stretched to cover every mistake made by a homo sapiens, the phrase loses all descriptive power.

So, to clarify - all errors made by humans are "human error" in some sense, but that renders the classification meaningless, so we should distinguish between simple, common errors and more worrisome ones by using words other than "human error". Your example of a programmer that did not foresee a future desired use could be referred to as a "requirements error" if the programmer reasonably could have or should have foreseen it, as "operator error" if the person using the program is meant to be trained to understand its proper uses, or a "misallocation of resources error" if the problem exists because not enough budget or time was assigned to keep the program aligned with current necessary uses.

I sometimes feel that a fair number of disagreements or disputes in the comments of Hacker News are best resolved by checking whether we are using words in exactly the best way.

It's human error, but it's not a bug. There's no mistake being made by a human due to faulty thinking or execution. You can have a good faith misunderstanding about what an author's intention, especially if your tool does what you want, and it's author decides the lib is headed in a different direction during the next update.

Not your fault, not the author's fault, just two different intents.

I think they need to further explain what exactly happened than simply stating "human error". If it is avoidable, admit what it is and also let other people to learn this lesson.

What configuration error (technically, by definitions from some IEEE guidelines, mistakes) did they make to have this happening?

No. I don't think they do. They've made a mistake and as long as they are willing to take responsibility for the consequences then I don't see the need for a public post-mortem just to satisfy our curiosity.

I guess you're right. Anyway the reason is pretty clear enough after actually reading the pasted code.

> What configuration error (technically, by definitions from some IEEE guidelines, mistakes) did they make to have this happening?

They edited .php file on a production server and made a typo, so the file, containing sensitive information started with "i?php" instead of "<?php".

My guess that happened because someone used vi/vim and hit "i" twice.

The fact that this occurred at all is still unacceptable, and we’ll be seriously evaluating and adjusting our processes to ensure an error like this can never happen again.

I'm not aware of any theorem provers for Apache config files.

Reply on news.ycombinator.com