Hacker News Reader

Comment by mechanical_fish | Hacker News Reader

Don't be too disappointed if a single submission gets a lukewarm or confused response on HN. The upmods and comments on here are a lot less consistent than what you're used to. ;) Just keep writing. It's really valuable.

Also, it's clear to me why your daily routine might sound like science fiction to the median HN reader: A lot of programmers have never seen a system like this. As those of us who were online during a specific half-hour period a couple weeks ago can attest, even Google doesn't have a system that's remotely as reliable as this: It appears to be possible to break all of Google search, worldwide, in ten minutes by misplacing a single character in a text file.

View on HN

Hmm, on your Google point, we know that they use partial-cluster deployments extensively, and several presentations point to sophisticated testing of these momentary guinea pig users. I wouldn't hold a one-time lack of a sanity check against their total uptime history. Tests ain't perfect.

I agree that we shouldn't extrapolate too much from this one incident. But it's not like Google's super-secrecy policy gives us much choice. If anyone from Google wants to tell us about their deployment infrastructure and explain why this one incident really was a nigh-impossible black-swan one-in-one-billion-hour freak of nature -- or why Google has sensibly traded away a certain amount of uptime in exchange for a more flexible architecture (or, perhaps, more cash to spend on tasty gourmet pizzas) -- I'm sure we'll all listen with rapt attention. Until then, we get to tease them mercilessly. ;)

Meanwhile, I'm sure that the original submitter would agree that tests ain't perfect. If you read the link at the top of this blog post:

http://timothyfitz.wordpress.com/2009/02/08/continuous-deplo...

...you'll find that this isn't merely an article about automated testing. Automated testing is just a part of the mighty continuous-deployment ecosystem being described here. It isn't even the real heart of that system: The heart is a planned, well-designed, semi-automated routine for rolling back changes in production. They roll out a change to a subset of their servers, monitor for statistical anomalies in the usage patterns of real, live users, and only continue the rollout if there are no anomalies. If they run into trouble, back they go.

I don't want to defend Google per se, but their uptime results speak for themselves. I don't see how a rare bug necessitates mocking.

And I agree about resiliency of the deploy -- it's what I meant by sophisticated testing of these momentary guinea pig users. Google's presentations on this stuff are about analysis and data gathering of changes both for immediate functional snafus and user preference for changes. i.e. probably state of the art in this regard.

> As those of us who were online during a specific half-hour period a couple weeks ago can attest, even Google doesn't have a system that's remotely as reliable as this: It appears to be possible to break all of Google search, worldwide, in ten minutes by misplacing a single character in a text file.

How can we conclude that the same isn't true of IMVU? The fact that such a rare event hasn't happened to them yet tells us very little.

I definitely wasn't disappointed. In fact it was refreshing; I didn't realize how hard of a problem other people considered Continuous Deployment (and in fact, how little other people had considered it at all).

I also didn't mean to rag on news.yc, on average the responses here are better than the original posts which is an unparalleled level of quality.

I left the technical issues unspecified in my first post on Continuous Deployment, and the comments on that post already had started discussing the path of solutions we ended up building ourselves!

I still have another post to write about this, because we also ship a native windows client. We ship daily prereleases of it, and roughly biweekly full releases (offered to all users). That's close to, but not quite as impressive, as the update system that google chrome uses. Google definitely still has us beat in certain categories.

lukewarm or confused response

I have often posed questions here about things I've been doing for years just to see what others are doing or if there is a better way. Invariably, someone tells me it won't work when I already know better.

This tells me 2 things. I've encountered someone who speaks when they should be listening (what else is new), and, more importantly, that I'm pushing the envelope enough to make otherwise knowledgeable people uncomfortable. Good.

Reply on news.ycombinator.com