tldw; He says he doesn't advocate it but they get away with it by having the community test it out for them in their meta site. Then the community writes up the bugs.
I take this to mean that he feels that StackOverflow doesn't need tests. Not that tests are useless.
- Stack Exchange employee
Pros:
* Tests are self-updating. Add a new feature: tests come in for free. Change a feature: tests automatically update. Fail to document a change: tests fail.
* Tests are unusually thorough
* Eventually consistent testing. If nobody ever complains, it probably wasn't a bug worth fixing.
Cons:
* Tests cannot be run offline. Feature must be committed and deployed before tests can be run.
* Potentially large quantity of false positives (bad bug reports)
* Potentially large quantity of false negatives (nobody notices particular bug, release considered good)
* Does not work for non-user-visible features
So basically you trade the reliability of your tests for a substantial build/release speedup. Some users experience each bug, but they are the users who are actively using the meta-community and have signed up to experience more bugs. Still, lack of pre-release unit testing must radically increase the importance of VERY careful code reviews.
Not the decision I would have made, but definitely has the sorts of advantages that a small team of engineers drool would drool over.