Comment by pmontra | Hacker News Reader

Comment by pmontra | Hacker News Reader

I agree that it is no synchronization primitive and that even if it was, it's very fragile. However I found it in nearly every project I worked on, in backend tests, frontend tests and even in integration tests where the library driving the browser should take care of waiting for elements to appear on screen. Modals and browser dialogs are particularly good at breaking stuff.

The dynamic is: developers write the the test and it usually works on their machines. If it does not there is a chance that they figure out what's wrong and rewrite it in the proper way. Tests often work locally, where the CPU and disk are nearly 100% available to run them. Sometimes they fail on CI systems. Developers scratch their heads and attempt the easy fix of adding a sleep of 1 second. That almosts always work and it took 5 minutes, code, commit, test run. They know that it stinks but they have stuff to do. Every few months somebody attempts to remove some of those sleeps with mixed and often unsuccessful results. Integration tests are particularly nasty.

As an intern I took a look at an unloved integration test suite that took something like 8 hours to run. Most of it was resulting from scattered sleeps waiting for Apache Solr replication, so I built a hack that queried Solr to determine the timestamp of replication vs the I think the committed document's timestamp (iirc, it's been a while) and waited for it to be observable, which cut 6 hours off the test's runtime.

Where on earth do people find interns this is good

I have a test suite with 100s of tests written by someone who was a big fan of this method.

When you run the test in isolation, 1s wait here and 2s wait there is not a big deal. But when you run 200 tests sequentially (because some of them don't work when they run in parallel), then it starts to add up, and your test suite starts feeling sluggish.

I'd be asking several questions here:

Why wait in the first place and how can I fix the issue of tests not parallelized

Both should be red flags in a codebase

I mean, there are some valid reasons why the tests are the way they are, and it's not easy to fix them.

Re waiting: Test are integration tests that wait for some event. Example: test changes a file on disk, waits for UI to update. We rely on the OS to report file system changes. We don't know how long it takes, because it depends on system load. We could just wait for the next file system event, but we might receive multiple file system events, and since they are processed asynchronously in our app, we don't know how long it takes for the UI to update. We could add some kind of notification when the UI updates to speed up tests, but that would mean adding extra code that might slow down the app just for integration tests.

Re parallellizing: Tests are integration tests that run against different databases. Not all databases have perfect transaction support, so sometimes one test interferes with another (eg. one test creates a table, another test fails because that table exists). So parallelising these tests would require multiple copies of each test database, which would cost more money, and spreading tests over multiple test databases would make the test execution code even more complex...

and thus you end up with slow software

Slow tests.

Yes, though is suspect it is cultural. A team that writes slow flakey tests probably writes slow flakey production code too, for the same reasons.

I'm sure that's a factor, but having slow tests makes it much harder to notice when you have made the actual code slower.

A team that insists on high-performance tests probably ships lots of bugs, because the tests are so local/isolated/mock-driven that they don’t actually exercise interesting interactions.

Reply on news.ycombinator.com