Ask HN: How often are you frustrated with flaky tests? | Hacker News Reader

Ask HN: How often are you frustrated with flaky tests?

3 pointsublaze6 comments

Are you looking for better tooling to debug/catch/auto retry flaky tests? What kind of tooling do you already have, and what alternatives have you looked for?

Is there some tool/service you wish existed?

View on HN

I've worked somewhere before where there have been flakey tests and they had a system to detect them and list them, and they could be assigned to people. Pretty nice.

I'd like it to exist as a service yes, if it can plug into any testing output, and accurately know that yes something is a flakey test. The telltale sign is a retry on the same branch fails then passes with no source code changes.

I am not sure however if any company I worked for would PAY for such a system. That's a different question. But as a developer I'd like to use one if it is well honed.

Just fix them? Where I work we explicitly do not automatically retry failing tests. Otherwise the incentive goes away to fix them. At some point somebody will be frustrated enough to fix the issue.

Note: we have run hundreds of little browser-based tests, visual snapshots and thousands of unit and integration tests on each PR. Flaky tests get annoying real quick and get fixed quickly as a result.

Write tests to be deterministic from the very start (mainly: learn to use synchronization primitives to synchronize events rather than trying on timing coincidences) and then you'll realize there's no such thing as a flaky test.

Have you written Selenium tests before, pointing to a staging or production build?

Yes, I have and it's IMHO a very poorly designed API (or rather, abstraction) for reproducible tests, but yes, the premise still stands. Instead of crossing your fingers and hoping a navigation has occurred/completed, test deterministically for the current state (inspecting browser navigation state, page url, and window events) to decide whether a certain action can be taken or is valid at the time. Basically, you should build your own framework upon which real tests can be run.

not to be too dramatic, but they are bane of my existence

Reply on news.ycombinator.com