This is your pilot speaking. Now, about that holding pattern...
googleblog.blogspot.com
googleblog.blogspot.com
The twitter fiasco of first removing a feature because it was "confusing" [according to them] and then backtracking and saying that it was removed because of engineering problems was interesting to watch.
For me it's always that same stupid routine operation that I've done 1000x before, so I turn my brain off while I do it. Then there's that moment of shocking clarity where you think "I didn't just... did I?" followed by the mad scramble to check and then the sinking realization that "yes, yes I did." shudder
http://nanog.org/meetings/nanog44/presentations/Monday/Gill_...
Most people clearly understand the benefits of adding process, but very few seem to realize the costs.
If I tried hard I am pretty sure I could create a checklist with 1000 items for each developer to go through, and no one could argue against any of the items - individually, they would all be ok / necessary / correct. However, if I forced every developer to go through the list every time, for every change, they would - rightly so - feel crushed.
With very few exceptions - where a new process is really granted, I see people trying to substitute either thinking or automation by process. Which is a recipe for bureaucracy, and in my view a good part of why working for a BigCo can be so miserable sometimes.
A new process should be a last resort, after we answered yes to: a) Is it really beyond us to automate this?, b) Is there some flaw in human beings that will ensure this will repeat? c) Are the consequences of this mistake really serious?
Surgeons are very very expensive, so their time is too.
Surgical instruments, on the other hand, are easy to replace ;)
An hour of Google's revenue lost from 14% of their customers costs them about $350,000 (judging roughly by Q109 revenue numbers). Had it been 100% of customers impacted during that hour (i.e. a bigger goof-up) they'd have lost ~$2.55 million.
If you had 9,000 engineers making 100k a year on average, they could each spend an hour every day on paranoid safeguard processes and only cost the company $308,000.
So is it worth investing in processes to avoid that? Absolutely. Even if they can't find a good way to automate this, it's hard to argue against protecting against that sort of loss of revenue however you can.
Uhm, it's closer to $500k. Per day. So based on today's goof, which is presumably rare, with your batch of added paranoia they'd be still be out $150k for the day, and they'd be out $500k all of the days that there wasn't a colossal screwup.
I wanted to search for something during the downtime, and I didn't go to Yahoo--I waited. They definitely lost revenue, but it is a ridiculous, baseless claim that everyone went somewhere else during the downtime. You have absolutely no data to make any such claim.
Furthermore, your calculated cost of an engineer's time is simplistic and inaccurate. It doesn't count the lost revenues from delaying the release of their work, and the reduced value of that money by gaining a lower time value for it (getting money earlier means more time to multiply it through investments and reinvestments).
And, even worse, you are comparing the DAILY costs of developer time (which you grossly underestimated) to something that happens, maybe, once or twice per decade.
It could cost them many millions--maybe even hundreds of millions or more--per year to implement such a policy.
That's why these mistakes happen--it is cheaper to fix the rare screw-up than to waste too much time checking everything, except for very few circumstances.
They respond awesomely, but they're not immune to major issues and I doubt it will get any easier for them.
However, the reality is that Google has positioned themselves such that they have quickly become a utility. This goes beyond search, they are a communications network (email, chat) and an ad service among other things. How many websites depend solely on Google AdSense for their revenue?
Just like electricity, telephone service, and other utilities, we have come to depend on them for living our daily lives. Loss of service at a large scale cannot be tolerated.
This may have been a relatively minor event, but the point remains that it still seems much too easy for such events to take place. I feel that Google should be one of the companies pushing technology in this area forward. Hopefully they will release information on how they plan to prevent this sort of event from occurring in the future.
You could theoretically say "at time T1, everyone save your current tables" and then sometime later (when you were confident it had propagated everywhere) say "at time T2, revert back to your tables as of T1" but you'd have to assume that no devices had joined or left in the intervening time; in practice the only way back is to trigger a new computation and return to an approximation of the earlier state.
When you make a routing update, it takes place immediately (generally). You can only undo it by making another update (presuming of course that your first update didn't impair your ability to get more data through...).
There are tools, like WANDL, that allow you to model your network and see what various changes or outages will look like in "real life". Of course, these simulations take time to setup and run. And, I've made simple routing updates before. I KNOW that this will work and I don't need to model it...
Back in the dot-com days of '99, I built and maintained an app that was pretty much identical to what Google Maps released 5 years later (yeah, with tiles and javascript dragging & all that). The downside is that the backend was pulling from this crap GIS system that was always overloaded.
So it would bog down under load. And it was my problem. And it hurt.
So 10 years later, I run a little site built on the Google Maps API. Today it bogged down under load. And it was sunny out so I went out rock climbing because a whole team of smart people were scrambling to fix it for me and it wasn't my problem.
Sometimes "not my problem" can be pretty nice.
I always wanted to pitch the idea of having a YC meetup at a climbing gym.
I seem to find myself in the 'states for about 3-6 months most years, usually in LA. If there were a hacker crag session when I was in town, I'd be there.
A climbing meetup is something I've been planning. I will probably ping the local (nyc) folks to see if they're interested.