Hacker News Reader

Comment by solatic | Hacker News Reader

If you have so many flaky tests, and their flakiness is so intractable, that you actually need to come up with SLOs for handling the flakiness to negotiate being allowed to address the flakiness in the test base, then quite frankly, you should be looking for another job, one where you can actually go ahead and just fix these things and get back to shipping value to customers, instead of one where you play bureaucratic games with management that cares neither about craftsmanship nor about getting value out the door as quickly as possible.

View on HN

This is a bunch of wishful thinking...

Some software projects, let's call them "integration projects" use third-party software they can do nothing about. And it just doesn't work well. But you have to use it in testing because... well, you are the integrator. The users have already accepted the fact that the software you are integrating doesn't work well, so, it's "all good", except it makes it very hard to distinguish between failures that need to be addressed and those that don't.

Just to give you one example of this situation: JupyterLab is an absolute pile of garbage in terms of how it's programmed. For example, the side-bar of the interface doesn't load properly quite often, and you need to click on "reload" button few times to get it to show up. Suppose now you are the integrator which provides some features that are supposed to be exposed to the user through JupyterLab interface. Well, what can you do? -- Yes. Nothing. Just suck it up. You can manipulate the threshold for how many times you will retry reloading the interface, but you absolutely have to have a threshold because sometimes the interface will never load (because of some other reason), and you will be stalling the test pipeline if you don't let this test fail.

In general, the larger the SUT, the more "foreign" components it has, the harder it is to predict the test behavior, and the more flaky the tests are.

But this isn't the only source of test flakiness. Hardware is another source. Especially in embedded software that has to be tested on hardware that the software company has limited access to (think something like Smart TV, where the TV vendor provides some means of accessing the OS running on the TV set, but they deliberately limit the access in such a way as to prevent the SW company from getting access to the proprietary bits installed by the vendor). So, sometimes things will fail. And you wouldn't know why and wouldn't be able to discover (as in, if you tried to break into vendor's part of the software, they'd sue you).

It sounds like your integration architecture is a mash-up. It is indeed quite difficult to handle external systems that are so tightly coupled to your own.

For standard server-server integrations, where the coupling is limited to an (hopefully eventually) well-documented API surface, it's much more straightforward to replace an external service with an internal mock.

For CI purposes, we eventually develop a more-or-less complete internal version of every external service, quirks and all. It's fun to see their bugfixes show up in our git history in behavioral mocks.

We don't put much emphasis into automating integration testing. We've found that with all the vagarities of flaky external services, it's usually necessary to have a human in the loop somewhere for integration tests.

> more-or-less complete internal version of every external service

Imagine what an integrator who works with public cloud would have to do?

Now imagine that integration with three different public cloud is only a small fraction of what our software offers...

Our clients integrate with all manner of public and private APIs and this is how we build every one of them. The flakier the provider, the more the gambit pays off.

You don't have to build the whole API, just the parts you use. And you don't have to implement the real service at all. It amounts to just a bit more overhead the first time you use a new API, but that's dwarfed by the amount of time your team spends learning how the thing "really works" anyway.

> For example, the side-bar of the interface doesn't load properly quite often, and you need to click on "reload" button few times to get it to show up... Well, what can you do? - Yes. Nothing.

So... go debug the JupyterLab code. Or open a bug with upstream. Or talk to their support. Or rant on Twitter to try and mobilize upstream to fix their shit. There's not nothing to be done, that's defeatist nonsense. In the meantime, in your codebase, mock the stuff you can't control and move on. That's why mocks exist, so that your test suite can be fast and deterministic even when speed and determinism are otherwise systematically difficult to achieve.

> hardware

Hardware is a totally different ballgame. And not one where you're trying to come up with SLOs for flakiness in a CI system. You don't continuously deploy hardware. You look over the test results, make a manual decision what needs to be retested, and just retest that before deploying. In dev, hopefully you have some kind of emulated device to develop against, in which case, the point is moot.

> So... go debug the JupyterLab code. Or open a bug with upstream.

Users want this particular version of JupyterLab. Not the one in the future which might get fixed. In the case of JupyterLab, I can maybe try to patch it and promise users that "it's the same but with the patch", and maybe they'll take it, but maybe not, because their IT vetoed only version X, and my X+patch is not exactly X.

But, it could as well be closed-source third-party software (for example, we help users to install PBSPro, EG and LSF workload managers). It's illegal to debug them or to patch them...

> Hardware is a totally different ballgame. And not one where you're trying to come up with SLOs for flakiness in a CI system. You don't continuously deploy hardware.

So much nonsense here...

* CI isn't about deployment.

* Yes, when I worked for a hardware company, we continuously tested integration between hardware components.

* Why wouldn't the concept of SLOs apply here? You say it without giving any explanation.

Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale.

A 1 in 10,000 failure can be a daily annoyance for your users even with just 100 daily active users who each make 10 actions on your app. At “internet scale” a 1 in 10k error frustrates a user every few seconds.

If your tests are so flaky that you need SLOs … your poor users …

> Every flakey test is a production failure that really happens to some users sometimes

No. Tests can be flakey when the code they're testing is not. Every flakey test is not a customer problem.

Training developers to ignore flakey tests also trains them to ignore real failures. Potentially many of them. Any of those might be real customer problems.

This! Flaky tests become a customer problem when developers learn to ignore them! You gotta weed them out.

> Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale.

Not really, the one I’ve seen most often is a shared resource failing - for example a Gitlab Runner not handling a new VM for a DB so tests fail etc

Also, dirty test harness state not being cleaned up between tests/suites, and threading issues in the test code

Depending on the problem domain, a lot of times the flakiness is in the way the test is written and not the code under test. You have to judge the relative cost of tracking down and solving that.

I don't think the point of SLO = flakiness out of control. The point of framing it as SLO is the realization that neither extremes are good. Flakiness cannot be allowed to get out of control that some efforts must be spent to contain it, but it's also unnecessarily perfectionist and thus the waste of precious engineering bandwidth to eliminate them completely. The whole point is to avoid "bureaucratic games", as you call it.

My theory is that the lack of easy mechanism to measure the flakiness is stalling the progress. If the overall flakiness can be measured, and the top offending tests identified, then I think it becomes no brainer to spend efforts curtailing them back when the flakiness gets too high, but otherwise exclude flaky tests from, say, PR merge gate.

Reply on news.ycombinator.com