Comment by cweagans | Hacker News Reader

Comment by cweagans | Hacker News Reader

Parent Full thread

4 hour tests are a nonstarter, but ponder this: how can you guarantee that bug will never happen again? You can be pretty sure. You can reason about the code that you’re now familiar with and infer (based on your current understanding) that something like that won’t happen again. But you can’t guarantee it.

Some bugs don’t really deserve tests that run all the time - that I’ll grant you. I frequently have tests for stuff like that as a form of documentation for myself down the road. A year from now, will you remember the specific steps you took to fix your case sensitivity bug? I wouldn’t. But my VCS will. Future me will be very grateful if I write a “verify this annoying case sensitivity thing isn’t a problem” script somewhere so that if I’m seeing weird behavior in the future, I can be reasonably sure that it’s a different problem (or that my fix got reverted or overwritten).

I also don’t find a lot of value in unit tests or having 100% code coverage. But that doesn’t invalidate automated testing altogether. I cannot tell you how often tests - even just simple end to end, happy path tests - have saved my ass.

I’m willing to bet that you’re already testing your software manually, right? Like if you change (for the sake of example) a form, you’re probably going to submit that form a few times with different inputs. Somebody else down the road is going to do that too. Why wouldn’t you write down the different inputs you gave so that the next person has them available? Maybe that person doesn’t have the same context that you do. Maybe they just aren’t as thorough as you. Either way though: if you’ve already done the work to figure out how to fully exercise the form, why would you make someone else do that work again?

The next logical step after writing that stuff down is to automate it if for no other reason than to avoid having smart, expensive people sitting around doing data entry when a computer can do it for them.

That seems extremely uncontroversial to me and I have a hard time understanding why someone wouldn’t be on board with that.

> But you can’t guarantee it.

The fundamental logic error in the justification for tests is that I don't need to guarantee that the issue will never reoccur. A reasonable level of confidence is more than sufficient for most projects, most of the time.

As an example of an environment where automated tests are critical: I read a post by someone complaining that when they worked at Oracle the automated test suite ran through hundreds of thousands of individual tests and took hours despite using a huge farm of servers.

That guy was wrong! I would absolutely test something like a commercial RDBMS to death. Those tests are not optional. Similarly, if I was developing a file system such as ZFS, I would also test the heck out of it. Famously, Sun had a test lab where robots would physically pull drives out of servers!

But would I test a typical web form with automated tests? No. It's boring CRUD code. It's going to work, because I use parametrised queries using strongly-typed ORM to a CotS database platform using "boring" code paths that are well known to work. If the DB schema changes, the build will fail.

I'm more interested in testing if the form UX layout "looks pretty" and has a "nice layout to help the workflow." Ensuring those will also exercise the boring parts of the back end code anyway as I click through the form a bunch of times.

Once the code has been established as working, and strong typing is used end-to-end, why bother testing it over and over?

Personally, I don't have time to spend on solving the same problem twice. I want a guarantee that something isn't going to break again. Yeah, it probably won't break again in the exact same way, but if you're writing your test to cover one narrow failure mode, perhaps it's time to brush up your software QA skills.

You're making a set of assumptions when you write a piece of code, right? All the steps between when you actually submit a form and when the data is saved in your database are assumptions that you're implicitly relying on in order for your application to function. Individually, each of those components can work just fine, but no matter how "boring" they are, your application can still be broken.

For instance:

> a distinct sort was case-insensitive but should have been case-sensitive. It was dropping a few hundred items out of hundreds of millions.

You don't know that it's _going_ to work until you've _tested_ that it works. Right now, it sounds like your testing is almost exclusively manual. That's a valid approach, but it doesn't scale very far (neither in terms of application complexity nor team size and composition nor project age).

> Once the code has been established as working, and strong typing is used end-to-end, why bother testing it over and over?

You've established that it's working as expected _now_. What happens if somebody submits incomplete or malformed or outright malicious data? What happens if somebody opens two copies of your form, fills them both out, and then submits each of them in sequence? Which one should take precedence and does the notion of submitting this particular form more than once even make sense? What happens if your database isn't reachable but you have data that you need to save? What happens if another developer on your team updates the ORM and that includes a change to how data is sanitized on its way into the database that breaks some implicit assumption about how your data is going to be stored?

I hope you have some way to catch that stuff in code review. Syntactic rigor and being a bottleneck in the code review process is only going to get you so far.

Personally, I don't want to be figuring any of that stuff out at 3am when the pager is going off. I want to be confident that the team has solved _and documented_ all of the potential problems that they can think of (in the form of tests).

Lot of good reasons above. Another is that if code has solid tests, it makes code review a lot easier / faster, especially if somebody else is modifying your code.

Reply on news.ycombinator.com