Hacker News Reader

Comment by kevingadd | Hacker News Reader

Caching test results sounds like a horrible idea, since the test could rely on external state or dependencies that weren't checked. For example, if you're testing something that opens a socket.

View on HN

People often come up with this absurd argument. If all your tests are opening sockets, it must be a nightmare as they may fail randomly every time you run them.

Keep tests that open sockets and otherwise interact with resources not controlled by the test environment itself separately, what most people like to call integration tests. Only those cannot be cached - and it's a huge trade off and while you get a lot of value from those tests, exactly because they can't be cached, you want to keep those to a minimum.

Most applications I've ever worked on talk to the internet or to other servers on the network, you can't close your eyes and pretend everything runs on one machine. You eventually have to test all of your software.

Certainly, you can optimize some of your tests by caching them! But they should be fast anyway, because your software should be fast. The software I currently work on has massive test suites containing around 50k tests that run in about 3 minutes per suite on my PC. It's time consuming, I suppose, but caching would mean that I'm not actually flushing out latent bugs. Instead, we run all of the tests every time and do things like randomize the execution order, so we find bugs before customers do.

At the point where your software interacts with threads (or the kernel!) you've already brought in non-determinism and trying to act like your tests are really deterministic is naive.

I've written so many tests in my life I think I know what I am talking about. It's not naive, it's what professionals do.

Learn to separate tests that use outside resources and behave undeterminastically from the ones that are never flaky, make sure to cache the latter. If you don't do that you're still in the dark ages of testing when tests are a pain to touch and fail all the time for no reason.

Many folks believe that builds and tests should ideally be "hermetic", and not depend on external state. This can make tests more robust, and facilitates tasks such as bisecting to identify the cause of regressions.

I'm a big fan of fast, self-contained hermetic tests, but at the end of the day you should be testing the actual behavior of your software. This means all your dependencies should be versioned so you can bisect actual behavior, not mocked test behavior.

I've also rarely ever touched software that is actually free of nondeterminism, so I am deeply skeptical of caching anything but the simplest test case. And those are fast.

You should avoid flaky tests. They slow down presubmits and make it hard to do bisections.

Sounds like you're testing wrong then ;)

How do you propose testing software that is meant to use sockets? Just don't?

Mock the entire socket API, so you're not actually testing your software?

At the end of the day you need to run some tests with external interactions, and you shouldn't cache those.

Also keep in mind any time you perform syscalls, you have external interactions. Their behavior could change and you want to know if they have intermittent failure scenarios. Using threads? You're nondeterministic, you should keep running tests.

> For example, if you're testing something that opens a socket.

Then the destination of that socket should be managed by the test too.

For example: https://pkg.go.dev/net/http/httptest#Server

Reply on news.ycombinator.com