Because race conditions are so hard to detect, the race detector is obviously prone to false negatives. Just because the tester doesn't find any race conditions doesn't necessarily mean that there aren't any. But the race detector never finds false positives. If it finds a race condition, that condition is very real.
You could just run your app with the race detector on all the time, but there is a performance cost to using the race detector.
One way to get around this in cloud/clustered environments is to deploy your app on a few machines with the race detector on and the rest with the race detector off. That way you're running your app with a production load and you're more likely to find race conditions, but you'll mitigate the performance costs associated with the race detector.
Running the detector on just a few nodes sounds like a great way to offset the performance penalty a bit. The docs on the race detector say that "memory usage may increase by 5-10x and execution time by 2-20x" which could be quite significant.
I also wonder about the effectiveness of randomly fuzzing your app with the race detector on as a form of testing.
Certainly for an entire program it is... not sure if you limit it to the just the code paths executed.
If you want a more guaranteed form of race freedom, you need to constrain what programs can do, either with dynamic restrictions on mutable state sharing (like Erlang or the parallel extensions to JavaScript do) or with a type system strong enough to reason about sharing of data (like Haskell or Rust).
https://code.google.com/p/data-race-test/wiki/ThreadSanitize...
Intended, yes. But that doesn't mean we also develop "cleanly&properly" at all times, in the real world. It's a great "backup" for when formerly-prototyping-now-production code is getting slightly out of hand over time.
The Go-documentation has a couple of great examples [1] that could easily happen in the real world (before the first coffee).