Waiting on Tests | Hacker News Reader

Waiting on Tests

etodd.io

47 pointset133724 comments

Something we did early on was to just get a rack of a dozen ci/test servers. These machines were really cheap since they can use consumer grade stuff. No UPS, no redundant PSUs, no raid, etc. Most importantly they are always there and always “hot”: most runs they can just git pull rather than git clone. They already have tons of intermediate results like node/nuget modules cached. There is a bit of maintenance but unlike maintaining production servers, maintaining these is pretty easy. If you screw up nothing bad happens. When they act up, it’s not extremely urgent.

> So in my opinion, the primary purpose of automated tests is not to avoid failures, but to speed up development.

Such a juicy premise, let's dig in. In my opinion the primary "speed up" to development comes in allowing changes - especially sweeping refactors - to happen without worry. As a corollary, they provide "primary source documentation" on the inner workings of the code so that less-experienced developers can approach such tasks without worry. The value of these are exactly the same regardless if your tests take 10 seconds or 10 minutes to run. The only difference is that we've sped up our critical path development work by not focusing on hyper-optimizing tests. And to start out the gate with such a premise but then fail to provide any data related to the effects on developer cadence, in a data-heavy writeup!?

This is fun. Seems like you did some good analysis.

I couldn't help but notice you run a "most common" test suite, implying that you have more tests that don't get run for whatever reason (the change doesn't affect that code, it's too slow, whatever). We end up having to do something similar to a small degree, and it bugs me.

What I would really like to see (and this might become a personal project at some point) is a way to optimize the tests at the step level. I believe this would accomplish several goals:

- It would reduce redundancy, leading to faster execution time and easier maintenance

- It would make the test scenarios easier to see and evaluate from a high level

Inevitably when you write tests you end up covering scenario X, then months or years later when working to cover scenario Y you accidentally create redundant coverage for scenario X. This waste continues to build over time. I think this could be improved if I could break the tests up into chunks (steps) that could be composed to cover multiple scenarios. Then a tool could analyze those chunks to remove redundancy or even make suggestions for optimization. And it could outline and communicate coverage more clearly. If formatted properly it might serve not only as a regression test suite definition, but also as documentation of current behavior. (Think Cucumber, but in reverse.)

Well kind of can imagine having 60% coverage where 10% is doubled because no one checked.

And there is no tool that looks for that really.

When I worked at the Research division of a large software company, one of my managers got upset when our unit tests (all of them put together) took more than 15 seconds to run, and he wanted to delete tests to get it back under 15 seconds.

He was simultaneously one of the smartest and dumbest people I've ever worker with. During that time, I did the best work I've ever done, and got the worst review I've ever had.

It was touched on at the beginnning of the article but something I find overlooked about having tests are:

- Ability of other developers to be productive on the project. Having tests tells other developers the intended behaviour and notifies them when they have broken it.

- Flow on effect of slow or large test suites. It demotivates developers to write new tests, run existing tests and accept failures if there are too many. This then leads to a lack of confidence, trust deteriorates between team members and development pushes towards going faster than being stable. Eventually, once enough bugs occur or a large incident, then you go back to looking at tests again.

If you have fast and reliable test suites, a developer wants to run and add to them. Developers feel that this is a high quality project that needs to be well maintained. This culture them permeates into other areas of your business.

From zero tests in 2019, to some 59k+ tests in just 4-5 years. On an established, significantly large product no less. Yeah I have zero faith in the quality and efficacy of those tests.

Yeah that's quite a few tests. I just assumed the author was summing individual assertions, or unit tests, or something.

The article is scarce on details about the size of the project and team. If the team is 100 people then having 10k tests written per year seems pretty normal.

But hey they have 100% coverage and lots of green text in their CLI. Surely they're shipping!

In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests. The benefit is preventing the introduction of new regressions unless they performed significant refactors along the way. And 59k tests is an average of 32 new tests added every single day of the week for 5 years straight. While it's easy to hit a couple dozen a day as an IC on the first month of an effort like this, the long tail is the hard part. Adding 32 new and valuable tests in a single day in year three of this effort sounds hard.

I'd be curious to hear about the manpower and logistical requirements behind this post. How many LoC was the original codebase? What's the current coverage at 59k? How many people were involved? How did they convince upper management to suddenly allocate this much opex spend for 5 years running, or what were they doing before this increase in testing expenditure?

> In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests

That's a great starting point for refactoring or just normal feature development. You need the confidence that your changes won't break the system in unforeseen ways.

Yes, I (as well as the commenter above, I assume) just doubt that the sheer number of tests will do this in a useful manner.

If you have a giant bunch of code that has been written without testability in mind, adding tons of tests mostly just locks down the interfaces of every single segment of code, making refactoring really hard to a point where the tests are useless because they have to be rewritten as well anyways.

Good test are written for good code that is written with testability in mind. In that case, only the relevant interfaces are proplery tested and not every little internal thing, which allows for proper refactoring and expansion without fiddling with the tests. This however also yields a lot fewer tests, which is why I doubt this large number of tests can be high quality and/or beneficial.

I tend to agree with you, although I'd say most of the tests are pretty "high quality" (for the most part they don't rely on global state and they run in parallel, randomized order). You pretty much perfectly described our current situation. There is a push to crystallize everything down to the last function. It's not uncommon to have one change result in hundreds of test failures.

> post-facto automated testing just locks in the codebase

Some of us call those pinning tests. It's pretty on the nose for your complaints, since it encodes a sense of FOMO right into the name.

> post-facto automated testing just locks in the codebase to behave exactly as it did before the tests

What, in your opinion, is the purpose of automated tests?

I can't answer for your parent, but for me it is the following. The main purpose of a test suite is to detect regressions in intended behavior. In a good test suite, if a test fails, then I can be sure that some part of the actual system would fail if I push this to prod.

Since the set of intended behavior is only a small fraction of all actual behavior, if your test suite pins down all actual behavior, rather than just the intended behavior, you end up with a very brittle test suite. That is to say, a test failing doesn't tell you "this code is broken", it just tells you "this code changed".

This is nuanced, but I try for tests that ensure the business rules (or product features) of the code are being exercised without enforcing specific implementation or style. Sometimes that does mean being highly specific, like when you are conforming to standards or specifications or providing a contract like an API used by third parties, but other times you don’t care how the sausage is made you just need to ensure sausage is getting made.

I don’t actually know how the article approached this but again in my own experience, when code is written without testing in mind from the beginning and you just go wild adding them later you instead get tests that enforce very specific coding styles, tests that enforce that the dependency graph is immutable, that internal data formats must never change, etc. If you can add tests without intimately understanding the business reason the code exists you’re more likely writing tests that just lock in implementation. This type of testing also means that every imaginable change will break dozens of tests and puts people in the mindset of just updating the test to pass instead of considering it a real problem because the tests stop giving meaningful information when they fail because literally everything you do will cause a slew of failures. That’s helpful in knowing the blast radius of your change but it multiplies the cost of change instead of reducing it.

My guess is that they use parameterized tests. With those, adding a dozen new tests takes only very slightly longer than adding one.

Here's the kind of thing I mean: https://github.com/simonw/datasette/blob/45b88f2056e0a4da204... - pytest displays that as 5 passing tests.

With hypothesis you can add 50,000 new "tests" in seconds :/

I dont think it's a very honest way of putting it though. It'd be more accurate to say that youve got one test and youve parameterized it.

Given the mention of git clone from scratch, docker image downloads, and empty go build cache, I'm assuming the author meant "terminate" when saying shutting off, and "creating" when saying spin up or cold start. Using ASG Warm Pools would help quite a bit in the scaling performance, as start/stop is faster than create/terminate and maintains cache state between uses.

If you want to optimize the create/terminate case, perhaps also create periodic snapshots of test instances after a run to populate the caches (or use something like Packer)

> Passed in 9m50s

The vast majority of build failures will not happen on the last test, and they also probably won't happen in the slowest set.

Running all of your tests in parallel to try to honor the responsiveness requirements of CI, is better than doing nothing. But consider that the information contained in a red build is much higher than the data contained in a green build. A build that goes red in 90 seconds is better than one that goes red in 10 minutes. And red in 90 is enormously more information than green in 10.

I am managing (among other things) a very large C++ library solving a very hard problem used by large scale enterprise software. The library has 9000+ tests. I haven’t had a bug in production for 5+ years. Despite making massive changes to the implementation details. I make changes and dump into production with zero worries. The productivity benefits of tests done well are amazing.

In my experience, turning on entire pre-prod infra to e2e test earlier has been effective at reducing overall test suite runtime (in that we're running e2e tests often to the extent we spot + eliminate the flaky ones). And by virtue of e2e tests being good at finding edge cases, there's less emphasis on unit/integration test performance

Reply on news.ycombinator.com