Hacker News Reader

Comment by qwph | Hacker News Reader

The stability of any system is roughly proportional to how much automated testing is done against it.

It should be, but usually it is just user base. Microsoft has the advantage of everybody sweating hard trying to make their things work. Their beta users are in the millions, and mostly are experienced users or even developers.

I see Linux developers talk about regression tests but couldn't find where is the central repository of this (is there one?) Regression tests IMHO should be distributed with the source code so everybody can run them. (I understand testing a kernel is not like testing any other user space program.) The BSDs do it, for example.

Or how much thought goes into it.

I remember reading about a study about a very different style of development from years ago, where automated testing was NEVER done, but instead formalized models of the program were created and mathematically verified.

If I remember correctly, the final result was a similar level of quality to an extensively tested system developed today, and the code tended to be of somewhat higher quality thanks to the extra thought put into it. (But I'm only going off vague memories here... I might be misremembering)

That's not an exclusive or though. If you have a proof of your code's correctness, I'd still like to see it being regressed against any changes to the code. Ideally the more different methods you have for validating that your code is doing what it's supposed to be doing, the better.

I'm actually quite disappointed to see my original comment with a negative score. I'm going to assume it was just badly phrased, as I don't seriously think that anyone believes that more testing of software results in a decrease in quality.

Of course it doesn't make software worse, but you make it sound like it's the only factor in software quality. Far from it, and IMO the degree to which it has become a religion is disturbing. I suspect that might be the source of the downmods.

I've seen projects with more tests than code, and I think that shows a problem with development methodologies. Simply reasoning about the flow through the code (and possibly using a powerful type system [think Ocaml] to help catch errors at compile time) would remove the need for many of these tests (not all tests should go, of course) and would result in tighter code.

Yeah, if you have more tests than actual code, that would seem to indicate something's gone wrong somewhere. And I'd agree that no amount of testing is going to turn an O(n) algorithm into an O(log n) one. There's no substitute for sitting down with a pen and paper and doing some old fashioned design sometimes.

I do think having a suite of repeatable test cases you can run against developing software is a useful thing to have, though. Not only can you test for correctness, but you can also run benchmarks against each modification to see if your performance or memory usage is going up or down.

It probably depends on what you're trying to do...

Reply on news.ycombinator.com