Hacker News Reader

Comment by titzer | Hacker News Reader

IMO thread pools are anti-pattern in several ways. They don't globally load-balance properly and can be exhausted. I think the Go concurrency model with many lightweight threads is really nice. The only issue I have is that race conditions in Go code can violate the type system and cause horror-crashes. I've never used Go in production but have talked to outfits who do and have confirmed that race conditions are a serious issue.

View on HN

One of my biggest pet peeves are race conditions in tests. They typically only show up when run in CI, and are impossible to reproduce locally. I have a hunch this is because of resource-starved VMs running on over provisioned hardware, which messes with time in unpredictable ways, making tests flaky. Sure, tests shouldn't rely on time, and these failures sometimes point that out for us to fix, but it's very annoying to troubleshoot when you can't reproduce it locally. I've tried everything from running in resource-limited containers to similarly spec'd VMs, and wasn't able to.

We could use a time-faking library, but sometimes this is unavoidable if some dependency, whether from a 3rd party or in stdlib, depends on reliable timing.

I'm wondering how other Go devs approach this.

Have you tried running your tests with the race detector enabled?

https://go.dev/doc/articles/race_detector https://go.dev/blog/race-detector

It only logs when a race condition happens, and it may have a non-negligible impact on performance, but it should print useful info to aid in debugging.

Oh, of course, I always run with -race enabled. But this is what I mean, the races are only detected in CI, not locally. Sometimes the stack traces point to a simple fix in the test itself, but often the issue is much more difficult to troubleshoot, especially since it can't be reproduced locally. It's also frustrating since this behavior is intermittent, and most of the time tests run fine in CI, but there are "troublesome" days when the issues happen much more frequently, which is why I think this is so dependent on the CI infrastructure (GitHub Actions FWIW).

A solution would be to run CI on self-managed nodes that behave more reliably, but that has maintenance costs that we don't want to deal with.

I see. My experience with detecting and fixing races using that tool, evidently applied to much simpler code, was basically getting notified that some variable was being written without synchronization by multiple goroutines, protecting it with a mutex, and being good to go. That's why I assumed linearity between running it and coming to a solution.

Sorry, race conditions? You really have to go out of your way to implement those.

Reply on news.ycombinator.com