Probabilistic Flakiness: How do you test your tests?
engineering.fb.com
engineering.fb.com
FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever.
So it should be no wonder that even tests are measured. Flaky tests get disabled. That's nice and keeps trust high in the test framework. The downside is that you are likely to forget to fix the disabled test, so now some functionality that was important enough to test in the first place becomes untested and leaves room for others to introduce regressions. This has happened in my team, but the pace at FB is so quick that there is no way to tie up all these loose ends without giving up your personal time. Or maybe it requires someone more diligent than me.
Either way, I just wanted to share that I personally found the idea of measuring test runs and acting on insights from thise measurements really powerful. It's a theme that can be found at all levels of FB infrastructure.
This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.
As far as libraries go, there is a mandate to use in-house patched versions of various sdks and libraries, for security/customization reasons. Otherwise, go nuts.
People write a whole lot of in-house software at FB, some of which is later open sourced.
> That just doesn't happen at Amazon
Oh please. We had a mandate to use Apollo.
PHP is banned at Amazon (whatever your view of PHP is, that is a top down directive).
The only freedom we actually had at Amazon was to use whatever open source software we could without contributing back to it (I'm sure there are examples where we did, ut not at the scale of FB).
And lastly, Amazon was all about telling me what to do. One day I got a phone call and I got moved from front-end app development to backend c++. Nobody asked me if I wanted to do it or if I would be any good at it.
FB spends months in bootcamp with teams trying to convince you to join them.
That's not a natural consequence of the structure.
If the top-down directives are issued by competent and financially accountable leadership, you get a nice efficiency pipeline.
How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature.
I have a few anecdotes where, as a Facebook group administrator of some large (tens of thousands of members) and small groups (few thousand members), there were so many bugs in moderating the group with an added mega bug where the “Report issue to Facebook” (or something to that effect) would also throw an error. Those bugs lasted years (going into the current year) before I gave up and quit Facebook.
Based on my experience over several years, I doubt there’s much rigor on many things. It may just be one of those marketing messages that have a little bit of truth and a lot of subterfuge.
Whether an internal team uses those tools effectively depends entirely on them and their expertise.
so this just means they've crammed more into your backlog than you can do. I would not give up any personal, unpaid time to do it, unless there's some promotion that you need which this is a demonstration/investment for.
I know what you mean though. Our deadlines are self-imposed, but the planning process is there to make sure that the effort remains high.
I believe the managers are responsible for squeezing out as much juice as they can in a polite way (as opposed to other places where they were pretty direct about it).
I suppose the combination of opportunity + compensation justifies the stress level and excitement at this point in my life.
I've worked at other places where people primarily coasted with low pay but received good retirement benefits. It seemed to work for them.
Ain't no one got time for flakey tests.
- They relied on something or other over the network, or - They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree.
The general trend I've seen is that the more "e2e" a test is, the flakier it is.
Selenium seems fine for the most part. It's a solid tool. Where it falls apart is when developers do not account for all the various ways the browser session will go wrong. There are simply too many variables at play. A/B tests, cookies, popups, network conditions, machine speed (and current load). Writing e2e is like being blindfolded and tying one hand behind your back while your coworkers take turns spinning you in your chair.
It constantly amazes me that companies think they can put more tasks on a developer's plate with zero impact to productivity. They really do believe that all testing is free and will somehow pay for itself. And yet a single QA human going through a simple testing plan will catch more actual bugs than hundreds of e2e tests that cost a literal fortune to maintain.
Tests I've found are much the same way. You don't write one flaky test and stop. If you write one and everybody is okay with it, you and your coworkers write more, and more, until there are 50, 100. Once the suite flakes out on an interval, nobody takes a failed test seriously, and then broken code doesn't get checked for hours. One broken test? I bet it's the usual. I'll just rerun it a couple of times.
The thread you're pulling on here starts to unravel the whole Continuous Integration sweater.
For testing purposes I often find myself making functions that take the date as an argument. If the language supports default values, I'll set it as a default value. If it doesn't, I'll make a convenience method or a null check to set it to 'now' if none is provided.
It turns out though that a lot of code we write to run within ±30 seconds of 'now' ends up over time having to run (or re-run) on old or future dates. So with the exception of logging and events, having that as an argument turns out to be useful or at least neutral.
For logging and events I'd probably use a mock timing library anyway.
> While engineers tend to trust passing test results, they often retry failing tests a number of times on the same version of code and consider failures followed by passing results as flaky. We do not have a good theoretical explanation for why this behavior prevails.
which seems to me like standard confirmation bias.
When writing a test or updating code, you clearly assume you got it right. So a passing test “confirms” that.
Personally, I find myself quite susceptible to this bias when updating code (i.e., when I haven’t broken any tests, I assume I updated my code correctly) but I’m much more cautious when writing new tests. I usually prefer to write a failing test, and get it to pass, to mitigate some of that bias (but fully admit that once it turns green, I’m overly confident at the “clearly correct” nature of the output).
Yet again, a problem is not solved but paved over by a new level of indirection, and the house of cards grows taller.
They can't help it, each facebook engineering blog post has to start with that scale porn.
Also, ideally your code and test are somewhat independently written, whether it is whitebox or blackbox. If that is the case,you greatly reduce the chance that both the code and the test are wrong in such a way that the test still passes.
This is why writing unit tests for stuff like getters and setters is really dumb. It doesn't really simplify checking whether the code is correct unless you use something like meanbean and thus provides some degree of separation between the test and the implementation.