Hacker News Reader

Comment by colomon | Hacker News Reader

And of course, the testing for any bug can be easily automated. Not!

Look, some things are very easy to test in unit tests: Does sin(180) return the right value? If I do operation X on supposedly const object O, does O have the same value before and after?

And some things are incredibly hard: Does this MP3 sound right after compression? Does the typography for this combination of letters look right? Is the user interface responsive enough? What about the crash that only occurs in just the right hard-to-duplicate circumstances?

Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort?

View on HN

> Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort?

It's probably for the same reason that evangelists for certain programming styles will dogmatically insist that you shouldn't have more than X lines of code or Y levels of nesting in your functions, without reference to things like the nature of the programming language or the essential complexity of the algorithm to be implemented. Inexperienced people tend to overgeneralise from their own experience, and those who have only ever worked in one small part of the programming world are the most disadvantaged of all.

There are lots of benefits from assuming you can do automated testing on code you write,

Testing makes you think about the specification, what it is you are actually writing and what makes it working.

It makes you thing about writing testable code, which means you worry about things like side effects, isolation and api's

and some times, it helps you end up with automated tests :)

testing doesnt just apply to mathematical notation, lots of real world applications suddenly become easily testable when you look at them with the eyes of a tester

> Does this MP3 sound right after compression?

A quick check for this would be to have a bunch of wav files that demonstrate different properties that the encoder is supposed to compress on, run them through the encoder, and then read them back out through a standard decompressor, and then verify that the waveforms match within a tolerance. With any release, you'll want to do some manual testing, but you can gain higher confidence on small changes between milestones.

> Does the typography for this combination of letters look right?

Same sort of thing, generate a bitmap from the output, compare against a pre-approved sample using a heatmap to determine differences.

> Is the user interface responsive enough?

Define what responsive means in quantitative terms. For most interactions, you're going to want sub 200 ms. For stuff where you want the user to wait, you'll have to define wait times ahead of time. Maybe you make changes that cause the latter to break because of a significant change in what you're doing, but it's good for people to know about that ahead of time, right?

> What about the crash that only occurs in just the right hard-to-duplicate circumstances?

Hey, nothing's perfect. But this objection is tautological. It's only hard to duplicate because you don't understand it yet.

"verify that the waveforms match within a tolerance" The problem is, that is not easy. Determining how close two pieces of audio sound is the majority of the lossy audio compression problem. You can of course check if you're output is bitwise identical, but when it isn't, you need ABX testing. With people.

Yes, it looks like manual test.

BUT, when QA team found problem, you can create automated test case, which will look for that particular problem only. Right?

For example, QA noticed "clicks" every few seconds in resulting sound. Is it hard to create automated test case for that?

I'm not sure how hard it would be to create a reliable test for clicks. Depends on the type of click, I suppose. Some would be fairly easy to detect (e.g., "for .1s, all samples output are 0, with loud samples on both sides"). Though I'm guessing that would actually result in plenty of false-positives, and would be a fairly carefully tuned (and thus fragile) test case.

A better approach might be to detect it in the frequency domain, after performing an FFT (that instant drop to 0 will generate a lot of energy on both sides). I suspect you'll still need the careful tuning; after all, a sudden burst of energy on your FFT could be a click, or it could be a cymbal.

Not sure how well this would work, I've never tried it, though it sounds like some fun code to write.

Why is it that these hard core test advocates always seem to assume that all bugs are of the former sort?

As someone who's more and more becoming a "hard core test advocate": not all bugs are the former sort! But why would that stop you from testing the ones that are?

For the record, I've written 18,000+ test cases in the last year. When you can write reasonable tests, they are an awesome tool for programming. I just object strongly to the notion that any possible bug can be reduced to an automated test. That's not my experience at all.

Reply on news.ycombinator.com