Hacker News Reader

Comment by Jenk | Hacker News Reader

From zero tests in 2019, to some 59k+ tests in just 4-5 years. On an established, significantly large product no less. Yeah I have zero faith in the quality and efficacy of those tests.

View on HN

In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests. The benefit is preventing the introduction of new regressions unless they performed significant refactors along the way. And 59k tests is an average of 32 new tests added every single day of the week for 5 years straight. While it's easy to hit a couple dozen a day as an IC on the first month of an effort like this, the long tail is the hard part. Adding 32 new and valuable tests in a single day in year three of this effort sounds hard.

I'd be curious to hear about the manpower and logistical requirements behind this post. How many LoC was the original codebase? What's the current coverage at 59k? How many people were involved? How did they convince upper management to suddenly allocate this much opex spend for 5 years running, or what were they doing before this increase in testing expenditure?

> In my own experience, post-facto automated testing just locks in the codebase to behave exactly as it did before the tests

That's a great starting point for refactoring or just normal feature development. You need the confidence that your changes won't break the system in unforeseen ways.

Yes, I (as well as the commenter above, I assume) just doubt that the sheer number of tests will do this in a useful manner.

If you have a giant bunch of code that has been written without testability in mind, adding tons of tests mostly just locks down the interfaces of every single segment of code, making refactoring really hard to a point where the tests are useless because they have to be rewritten as well anyways.

Good test are written for good code that is written with testability in mind. In that case, only the relevant interfaces are proplery tested and not every little internal thing, which allows for proper refactoring and expansion without fiddling with the tests. This however also yields a lot fewer tests, which is why I doubt this large number of tests can be high quality and/or beneficial.

I tend to agree with you, although I'd say most of the tests are pretty "high quality" (for the most part they don't rely on global state and they run in parallel, randomized order). You pretty much perfectly described our current situation. There is a push to crystallize everything down to the last function. It's not uncommon to have one change result in hundreds of test failures.

> post-facto automated testing just locks in the codebase

Some of us call those pinning tests. It's pretty on the nose for your complaints, since it encodes a sense of FOMO right into the name.

> post-facto automated testing just locks in the codebase to behave exactly as it did before the tests

What, in your opinion, is the purpose of automated tests?

I can't answer for your parent, but for me it is the following. The main purpose of a test suite is to detect regressions in intended behavior. In a good test suite, if a test fails, then I can be sure that some part of the actual system would fail if I push this to prod.

Since the set of intended behavior is only a small fraction of all actual behavior, if your test suite pins down all actual behavior, rather than just the intended behavior, you end up with a very brittle test suite. That is to say, a test failing doesn't tell you "this code is broken", it just tells you "this code changed".

This is nuanced, but I try for tests that ensure the business rules (or product features) of the code are being exercised without enforcing specific implementation or style. Sometimes that does mean being highly specific, like when you are conforming to standards or specifications or providing a contract like an API used by third parties, but other times you don’t care how the sausage is made you just need to ensure sausage is getting made.

I don’t actually know how the article approached this but again in my own experience, when code is written without testing in mind from the beginning and you just go wild adding them later you instead get tests that enforce very specific coding styles, tests that enforce that the dependency graph is immutable, that internal data formats must never change, etc. If you can add tests without intimately understanding the business reason the code exists you’re more likely writing tests that just lock in implementation. This type of testing also means that every imaginable change will break dozens of tests and puts people in the mindset of just updating the test to pass instead of considering it a real problem because the tests stop giving meaningful information when they fail because literally everything you do will cause a slew of failures. That’s helpful in knowing the blast radius of your change but it multiplies the cost of change instead of reducing it.

My guess is that they use parameterized tests. With those, adding a dozen new tests takes only very slightly longer than adding one.

Here's the kind of thing I mean: https://github.com/simonw/datasette/blob/45b88f2056e0a4da204... - pytest displays that as 5 passing tests.

With hypothesis you can add 50,000 new "tests" in seconds :/

I dont think it's a very honest way of putting it though. It'd be more accurate to say that youve got one test and youve parameterized it.

But hey they have 100% coverage and lots of green text in their CLI. Surely they're shipping!

Yeah that's quite a few tests. I just assumed the author was summing individual assertions, or unit tests, or something.

The article is scarce on details about the size of the project and team. If the team is 100 people then having 10k tests written per year seems pretty normal.

Reply on news.ycombinator.com