Hacker News Reader

Comment by k__ | Hacker News Reader

I talked to a guy at Mozilla once and he said they got a pretty instable build. Some tests take a few runs before you know they are really okay and don't fail because something is wrong.

Hope Rust will get this problem solved :)

View on HN

The build itself is stable. The testsuites, not so much. The problem is that written testcases sometimes unwillingly rely on timing or races. When you have thousands and thousands of tests, as Firefox has, there's a reasonably likelyhood at least one of them will trigger.

The tests are mostly written in JavaScript, the language Firefox needs to run, so not much Rust can do there.

To understand the constrains we have here, a ballpark figure is that we run a million tests per commit across all platforms (this is based on counting test files, so doesn't account for the fact that some files can contain thousands of separate test functions, but also doesn't account for the fact that we don't run every test job on every push). On average we see around 10 test failures per push due to test instability. So "on average" tests fail about one time in one hundred thousand. Of course, in reality there are some tests that fail much more often than this (one time in a hundred, say) and many that ~never fail.

Whilst test raciness is perhaps the most common problem, we also see intermittent failures due to race conditions in the browser code itself as well as through infrastructure instability. However it's unclear that moving to Rust will make much difference anyhow; Servo still sees intermittent tests so merely eliminating data races in safe code is insufficient to fix this problem.

This is a really interesting problem area. Tests that fails sometimes are really annoying because of the "broken windows" analogy. Are you using the most unstable tests as input of what to redesign next? Is it kind of a "deal with it" situation, where you need to retry the test suite a couple of times per commit, until it becomes green?

The test suite doesn't get retried, we just wait for another commit to come in; it's not generally a long wait. See https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound

Another problem is that running the tests takes ages.

You both "deal with it" and there's task forces that look at the tests that fail most often, and try to find someone with the skills to investigate.

How do you rule out that the races are a problem in the code vs a problem in the tests? If we have tests that are sometimes red, we strive hard to remedy them. We've found issues before where we thought it was the tests that were wrong when it was actually the code.

You can't easily, you have to debug them. Debugging intermittent test failures is hard, and you have no guarantee you're actually improving much tangible things despite the time invested in them. It just sucks.

Intermittently failing tests are a problem that all large code bases have, regardless of the used language. See for example this post about Google's code base[1], which is largely Java, Python, and to some extent Go and Dart, IIUC: https://testing.googleblog.com/2016/05/flaky-tests-at-google...

[1] They don't say which code base the post is about, so I'm just assuming that it's not Chromium. If it is, it'd largely be C++.

Reply on news.ycombinator.com