Comment by emmett | Hacker News Reader

Comment by emmett | Hacker News Reader

Parent Full thread

1) We've started introducing tests - as you grow I think they're essential. I would say though, not all bugs can be caught by tests (and in fact, I once went over the last 10 bugs we had fixed and discovered that none of them could have been caught by any testing framework). Here are some kinds of bugs you can't catch with testing:

- The code assumes that an external service is highly available

- The page looks glitchy in IE

- If you use that flash api, computers with an nvidia graphics card will show a green screen

- If you open and close a socket quickly, flash will hard-crash the entire browser...sometimes

- Under peak load, when a vacuum is running, this new query will hose the DB

That said, it sounds like this bug COULD have been caught by testing (though I don't know the details and can't be sure). And tests can be darn useful, when used in the right circumstance. But in the hierarchy of important things for running a production website, here's what I would list in priority order:

- Extensive monitoring and alerting

- Detailed, easily accessible logging

- Code review required and taken seriously

- Hiring good developers

- Automated deployment and rollback

- Integration testing

- Unit testing

We've been working our way down the hierarchy :-)

2) We have 100% automated deployment and rollback, and a neat internal tool called Brigade for managing it that I really should get around to open sourcing some day...

For what it's worth, the payment servers were one of the only parts of our system that did have (manual) tests in place at that time (because it took 24 hours to do a restart). The bug in question was in DB timeout code, and all tests passed (until several weeks later, when the DB had problems). Now, clearly coverage was insufficient, and I do agree that adding more testing (as we now have) is a good idea. But some bugs are always going to slip past. Code reviews are what could have caught this bug (which we also now take more seriously).

Every hotshot thinks they don't need tests until they try writing tests for a while, then they usually become evangelists.

My shock and amazement comes from the 24 hours to restart your payments system? Whaaaaat? If that is correct you guys have far, far more issues than not having tests.

How do you restart a system which holds open long-lived connections without kicking everyone off?

If you want to be kind to your users, you stop accepting connections on a new server, wait a period of time (the longer the better), and then restart. It's a matter of how many users you're willing to disrupt for how quickly you can restart the system.

The canonical example at Justin.tv is the video system: people broadcast for days to weeks at a time. If you restart the server they're connected to, their stream will be disrupted (even if the auto-reconnect works).

We have a separate system that handles most of the complexity which is stateless, but sometimes you need to restart the actual connection-holding-daemon itself. How would you suggest doing that without disrupting service?

Unit and Integration Tests don't always catch everything but cluster immune systems do a pretty good job on the important stuff:

When someone wanted to do a deployment, we had a completely automated system that we called the cluster immune system. This would deploy the change incrementally, one machine at a time. That process would continually monitor the health of those machines, as well as the cluster as a whole, to see if the change was causing problems. If it didn't like what was going on, it would reject the change, do a fast revert, and lock deployments until someone investigated what went wrong.

http://www.startuplessonslearned.com/2008/11/five-whys.html

Monitoring tends to be far more useful than tests for web apps. And despite how advanced this system sounds, it's usually far less work to monitor a web app than it is to test it.

Reply on news.ycombinator.com