Hacker News Reader

The benefit of testing is confidence that things are working. There are side benefits, like acting as documentation and examples, but they should not be the focus.

Writing tests (or deleting them, refactoring them, etc.) should always involve a cost/benefit calculation, even if it's a rough mental estimate that's not written down. In particular, that requires answering "How much effort will this test take me to write/debug?", "How much extra confidence will having this test give me?", "What level of confidence do I feel comfortable with?" and "Could I achieve higher confidence spending this effort on something else?".

This doesn't need much effort. For example, we might think "This array indexing took me a few attempts to get right; it could result in misleading reports, so I'd better throw a few edge cases at it to make sure I understand it correctly.". On the other hand we might think "This class doesn't have any unit tests, but there's already a bunch of validation on the results; it only affects the page layout, and it'll be obvious if something's wrong, so I'll leave it for now and put some extra checks on the gnarly payment system."

This approach makes sense regardless of the situation. In your startup example, it might make a lot of sense to write some tests that, for example, spider our site for broken links; or look for certain strings on certain pages. Things which are quick to write (they could even be shell scripts), but give us prior warning that our demo will go wrong. It would make less sense to write a pile of unit tests for the internals of some boilerplate piece of the system.

Code coverage, dogma, etc. should not be the deciding factors for what to work on.

>The benefit of testing is confidence that things are working.

No it gives you confidence that the test are passing.

I have worked on code bases that are well tested and code bases with non existing testing. The tested ones contain just as many bugs as the non-tested ones as far as the end user is concerned. In a year and a half working at the "well tested" company, regression tests caught an error one time. That was good but considering the testing team was bigger than the dev team, I question if it was actually efficient or not.

You misunderstand me. I never said that testing is an efficient way to gain confidence, or that tests give much confidence, or that only testing can provide confidence. Rather, I said:

> The benefit of testing is confidence that things are working.

Some people treat testing as a gamified metric (e.g. code coverage). Some people treat tests as executable documentation. Some people treat tests as some essential requirement, without which the software cannot be shipped. I'm saying that those are not good reasons to write tests. Tests should be written to give us confidence that things are working.

How much confidence will we gain? How much do we want? How much will it cost? Where should our efforts be directed? Those vary from project to project, and is exactly why I say thought should be given to the cost/benefit tradeoff.

(Personally I think automated testing can be an efficient way to quickly gain a lot of confidence in a system; especially if we do property checking of high-level functionality, instead of unit testing of low-level details. However, that's a separate point; we can disagree on that, whilst agreeing on the need for weighing costs versus benefits.)

Regression tests don't usually identify bugs in new features, unless they interact with the older code or required changing the older code. In my experience, they're useful to have, but not useful for identifying novel issues. I recall reading someone's analysis of applying fuzzing and that using a particular fuzzer would eventually stop discovering issues, but that didn't mean the system under test was perfect. It meant that they had worked out most (if not all) the errors that fuzzer could identify for them and needed a new fuzzer.

The same thing with regression testing. I wouldn't stop running regression tests just because they aren't identifying issues anymore, but I wouldn't rely on them to discover novel issues. That's what the new, updated test suite should be doing. A regression test suite will not have a test to cover a new feature. It will only make sure that the new feature doesn't break the old one.

[0] https://blog.regehr.org/archives/1796 - found it

> Regression tests don't usually identify bugs in new features

Neither do unit tests in most cases.

This doesn't seem to follow from my comment so I have no idea what your point is.

> To me there's still too much "testing is my religion" amongst developers

Maybe some developers. On the French web market: almost no one is.

> What's the return on investment on writing tests?

I only write end to end tests. Because I come mainly from a maintenance background: unit test for test coverage are a hindrance. But testing functionalities of an app? That's what a client want. They don't care about how you implementing things, only that it works. And that old bugs don't come back.

Yes it means new functionalities have higher estimates. But they tend to not come back once done. And doing it with end to end tests means you should easily be able to trash all the code and replace it with something else doing the same work.

The problem is the tooling. Mocking external APIs tend to be a pain even with Wiremock. Testing a GUI is not easy: a Sikuli server to drive a GUI with screenshots would be a boon. And it tends to be slow. But that's because most of the work has been going to JUnit-like tests for years.

> "What's the return on investment on writing tests?" doesn't get asked enough

I consider tests to be highest return on investment for velocity.

The smaller the change you are making, the more likely your tests will assist you in delivering faster software.

Sample sizing is probably biasing me here, but all of our more legacy code bases have higher costs for feature delivery to the point we have begun implementing logic at the API level (instead of the monolith) just to able to minimize the overall cost. The code bases that get features faster are more modern React code bases that have tests while the legacy code bases are Java or Scala with close to 3% code coverage.

> all of our more legacy code bases have higher costs for feature delivery

Compared to one another, the nominal cost of feature delivery in legacy is higher. This is a correct assertion from the perspective of the person maintaining those systems.

But from a business perspective, as long as the operational costs for that component don't exceed it's revenue / value generated, keeping legacy code running is perfectly valid.

There's an opportunity cost calculation at it's base: When does the compounded sum of all the time "lost" due to the nature of the code (lacking tests, architectural limits,...) outgrow the cost of decommissioning / replacing the legacy code? As long as the latter costs more then the former, it makes sense to keep the legacy code around.

Writing tests comes at a cost as well. The same calculation applies here as well. Is it strategically sound to sink time and money in writing tests if the costs of doing so outstrip the marginal gains on efficient feature delivery over the projected lifespan of a legacy component?

Maybe. Maybe a feature-rich but rough and unpolished prototype is what will delight customers the most.

Or maybe focusing on an absolutely minimal core, but executing it really well with high performance and having it run rock-solidly will attract the potential customers more.

I think this depends a lot on demographics, but in broad strokes, I think the former is overvalued compared to the latter. I.e. contrary to your experience!

I would be inclined to agree except people generally have low standards of performance (ofc depending on the industry!). People also rarely care about the hand of the puppeteer, what keeps the lights on parts especially if we're talking about a prototype.

I think people care a great deal more than they have words to express. We're not very good at educating our customers in ways to discuss and analyse technical performance. We're getting better at it when it comes to uptime, but latency discussions still lag (hah!) behind.

(This is from purely personal experience. I have only had one customer ever, when asked about performance requirements, who could list some. Everyone else has been "I guess I want it to be... good?")

The prototype serves to figure out what the customer does not need as much as what they do need. Polishing the wrong thing is a waste of time.

For me tests are essentially just a repeatable REPL session. And like a REPL they can radically improve my development speed.

So asking are these tests worth investment is silly, because they've decreased the time for me to write the code. It takes less investment.

I wrote a one-off test script to do a REPL session as tests. You just make a file with the commands you want, run it, and the system replaces the input-only file with the inputs+outputs. Subsequent runs replace the output.

Then you can check that file into version control, and whenever you run it you get a diff of the broken tests. If they're not actually broken, just check in the new file and you've updated your tests. It works really well for this project, a search engine.

https://github.com/pipedown/noise/blob/master/update-test-re...

https://github.com/pipedown/noise/blob/master/repl-tests/col...

Tests decrease development time!

Results may vary.

Yep, this is something I always try to instill in my peers when writing tests.

How much time does it take to get 100% code coverage, how much do you get paid, and if it was your startup would you burn through payroll cash to write them? When you start adding dollar amounts to things in terms of an engineer's salary, it's easier to see what is really important and what is waste. You are paid to write correct business logic. Tests are important, but only in that they verify your business logic is correct and stays correct.

I think in the situation you're describing, with a prototype being the difference between the business existing at all or not, you could justify all manner of sins. If you had a prototype already 90% done in COBOL it would potentially be the right call to finish that off - that doesn't mean using COBOL is the right thing to do in general. To use a common HN term, that is a situation where it is perfectly sensible to acquire technical debt, just as their are non-software contexts where it is financially sensible to acquire a bit of regular debt for a while. It doesn't shed any light on whether a lack of tests is technical debt.

(What you've described is a classic problem of junior devs trying so hard to do the right thing that they don't step back and ask why that is the "right thing" in the first place and whether that actually applies in their situation. I like to call this phenomenon the "overenthusiastic amateur". At least it tends to go away with experience - unlike the underenthusiastic dev who doesn't care what the aftermath is so long as their code works long enough for them to move on to the next thing!)

> "testing is my religion"

Not religion, but a necessary tool to knowing where to focus my attention.

> "here's the quantified value of me spending 3 days writing tests and here's how that fits in the context of where our business is right now"

How do I quantify 'getting the task done at all'?

EDIT: I mean it. To accomplish development tasks, I need to be able to run the code I'm working on and see that it works.

Are you actually suggesting that before I start a ticket, I should first do a quantitative analysis of the value of being able to do that? How?

  Imagine some incubating startup where you have an initial 4-week runway for development, with the goal of getting a prototype working

Are you talking about an actual prototype or an MVP? A prototype meant to showcase some idea doesn't need any unit tests. An MVP? probably yes.

Now back to the practicality of software development. Unit tests are fundamental when working in a team setting. What are people doing in PRs without unit tests?

> Imagine some incubating startup where you have an initial 4-week runway for development

Who have been asked to build something that does X, Y, and Z. Without tests how do they know when they're done? How do they convince other people that they've done what they were asked?