Hacker News Reader

Comment by vasco | Hacker News Reader

Skip your flaky tests should be a religion. There's nothing else I feel as strongly about regarding CI optimization. If a test is flaky, it gets immediately skipped. Even if you're working on a fix, it's skipped until you solve it, if you ever do. Most of your CI problems can start to be solved by applying this simple rule.

How do you know if it's flaky? You keep a count and any time a test fails/recovers 3 times it gets skipped, even if there's weeks between failures. You can make it more complex for little gain, but I've found this system will have teams actually prioritize fixing important tests, but mostly it has proven that many "important tests to keep even if they are flaky" never were actually important or end up getting re-written in different ways later on.

View on HN

I feel the same, I can't understand why people value more having a flaky test than not having it.

In both cases, the feature is not validated (the test failure is ignored), but not having the test is transparent (the feature is not being validated), improves pipeline speed and reduces the noise.

Maybe people just like the lie, that the feature is being validated.. I don't really understand.

This is indeed a religion, because in my experience people tend to feel strongly holding very different positions. You can already see it in this thread.

I think quantifying and prioritizing is key, like you wrote. Respected engineering organizations like Google and GitHub all came to the same place. Flakiness is often unevenly distributed, so find & tackle ones that are the worst. Don't try to eliminate the flakiness because that's not economically viable.

I'm trying to put my money where my mouth is... we'll see how it goes.

> You keep a count and any time a test fails/recovers 3 times it gets skipped, even if there's weeks between failures.

Congratulations, you now have an untested feature.

Flaky tests should be fixed, of course. But just deleting them isn't fixing them, and it leaves you worse off, not better.

May i ask how you feel about TODO comments?

If you re-read my comment you'll see I already addressed this. Not having a test is better than having a flaky test.

The team should feel the pain of having their shit test skipped without them being able to stop it and it's up to them to either fix it and bring it back or cope.

> Not having a test is better than having a flaky test.

What i'm saying is that you are completely wrong about that. Sorry if i wasn't clear.

Ok that's more clear, this is fine then we know we disagree (also why i called it a bit of a religion). Have a great day!

PS: if the test was flaky, you already had an untested feature

Or just fix them? A test shouldn't ever, ever, ever be flaky. It can happen because we all make assumptions that can be wrong, or forget about non deterministic behaviours, but when it does a flaky test should be immediately fixed, with the highest priority.

That is out of your control across many teams. Your only hope is having hard rules on what to do with flakyness so that teams aren't able to spend 3 months telling you "please don't skip because we'll fix it in our next sprint".

Fix the test or remove it.

I've often been tempted by the "delete your flaky tests" extremist sect of that religion :)

Reply on news.ycombinator.com