Where do Google's flaky tests come from?

You need to handle other problems (back-pressure, priorities, timeouts, ...) but if you need it you need it.

You can see it here:

https://docs.google.com/document/d/1mZ0-Kc97DI_F3tf_GBW_NB_a...

This assumption doesn't make much sense though. `y = x + 1` shouldn't flake. Nor is your comment here a correct reflection of "size". Size in the article means time the test runs or amount of memory used (ie. constructing a larger SUT), not LoC. Edit: I'm partially mistaken, one measure of size is binary size, which should be generally correlated with LoC, though not absolutely. Webdriver tests which use image/video goldens are an example where test size may not correlate with LoC at all.

>This doesn’t really make any headway towards understanding the problem.

The second part of the article does address exactly this (webdriver tests are correlated with flakyness, although they're also larger by virtue of needing to spin up a headless browser, which has a memory cost).

More edit:

As someone who has written and heavily deflaked some (very large) tests using, I believe, the "internal testing framework" mentioned in the article, flakyness always appeared as a tradeoff with speed. Parallelizing tasks introduces a level of randomness which can lead to flakes. Short(er) timeouts on things that can have long 95 or 99th percentile latencies, etc. It can sometimes be worth it to have a test flake some small percentage of the time instead of waiting longer (granted this is also often a flaw in test design somewhere).

But when you're testing a big system with many moving parts, it's not uncommon to have something that takes 30 seconds 95 or 99% of the time, and every once in a while 5 minutes. The question is if its worth it to keep extending your timeouts for n-sigma events, especially since that means that when your tests are actually broken, they'll potentially take much, much longer to complete. You can lower the false false negative rate, but the cost is that true negatives become much more expensive to compute. Is that worth it? It depends.

Of the tests I've seen, one usual source of flakyness is ignored side-effects between tests (DB is not emptied, some source of persistence is not reset - could be a 'global' variable or a mock library not mocking things correctly), sometimes import conflicts (some module clashes with a builtin library, but if you do this in a certain order it works)

Other sources include people using the equivalent of "now()" but then you're on those small time periods where your assumptions break (for example, now() is the last day of the year and you're assuming tomorrow is still in the same year, things like that)

> 4.2 million

I don’t know why the developers don’t care but I suspect they’ve learned to ignore specific tests which repeatedly fail.

Like in most development places management doesn’t give a flying shit about all of the tests passing if it means a feature will ship.

Probably they’re mostly older tests and so fixing them detracts from adding a new fancy feature

They’re hard to fix. If you can’t reliably reproduce then you don’t even know if you that it’s fixed or you’re just lucky.

They’re hard to find. The change that they first appear tends not to be related to the cause. Leads to a lot of “not my problem”

Really that’s just lack of ownership. Ultimately who owns the entire codebase. You say “my code” but I see a lot of people thinking that means the files I edit, not everything

someone has to explicitly decide to spend day(s) fixing tests, and then only might fix a couple. The roi is super frustrating and it’s hard to spend whole days on a single test.

There’s a ton of broken window theory. Once you get used to hitting rebuild, you just always hit rebuild and don’t even know it’s a new flake.

The most effective thing I’ve seen when you hit this point is get all the test data, junit xml say, somewhere queryable and make graphs and explicit parts of okr with time against it.