Beyond the most trivial of projects, I can’t think of a single project that I’ve worked on (or even looked at on GitHub) that has weighed the costs and benefits of automated testing and decided it just wasn’t worth it.
Beyond the most trivial of projects, I can’t think of a single project that I’ve worked on (or even looked at on GitHub) that has weighed the costs and benefits of automated testing and decided it just wasn’t worth it.
I'm dealing with a migration of a legacy code base right now, today, that could probably use some tests! But even suggesting this is a complete non-starter: The time required to write the tests vastly exceeds the time and budget allocated to the migration project. I'll simply enable an APM tool that costs $50 a month and call it a day. If something crashes in UAT we'll catch it. If not, we'll definitely catch it in PRD. If not, then it's probably not worth dealing with!
I've added tests to some of my projects in the past and caught virtually nothing with them. I always run my code "through its paces" at least a few times with a REPL, debugger, or some sort of tracing tool before it's ever committed to the repo.
The kind of bugs I find in production would never be found by typical automated test suites. Things like logic errors due to misunderstanding the requirements, or deadlocks in a database that only manifest with an obscure multi-user workflow combination.
As a random example, a significant issue in one codebase was that a distinct sort was case-insensitive but should have been case-sensitive. It was dropping a few hundred items out of hundreds of millions. Will your tests find this issue? What if I told you nobody noticed for years, and that the processing where this manifested takes 4 hours?
Will your dev team still be productive if changes require 4-hour tests every time they hit build or check in the code? To find a one-time issue that won't reoccur now that it's been fixed? Really?
Small, fast tests often find nothing and are worthless because they don't test anything of interest.
Big complicated tests are time consuming to write, still often find nothing, and slow down processes.
Some bugs don’t really deserve tests that run all the time - that I’ll grant you. I frequently have tests for stuff like that as a form of documentation for myself down the road. A year from now, will you remember the specific steps you took to fix your case sensitivity bug? I wouldn’t. But my VCS will. Future me will be very grateful if I write a “verify this annoying case sensitivity thing isn’t a problem” script somewhere so that if I’m seeing weird behavior in the future, I can be reasonably sure that it’s a different problem (or that my fix got reverted or overwritten).
I also don’t find a lot of value in unit tests or having 100% code coverage. But that doesn’t invalidate automated testing altogether. I cannot tell you how often tests - even just simple end to end, happy path tests - have saved my ass.
I’m willing to bet that you’re already testing your software manually, right? Like if you change (for the sake of example) a form, you’re probably going to submit that form a few times with different inputs. Somebody else down the road is going to do that too. Why wouldn’t you write down the different inputs you gave so that the next person has them available? Maybe that person doesn’t have the same context that you do. Maybe they just aren’t as thorough as you. Either way though: if you’ve already done the work to figure out how to fully exercise the form, why would you make someone else do that work again?
The next logical step after writing that stuff down is to automate it if for no other reason than to avoid having smart, expensive people sitting around doing data entry when a computer can do it for them.
That seems extremely uncontroversial to me and I have a hard time understanding why someone wouldn’t be on board with that.
The fundamental logic error in the justification for tests is that I don't need to guarantee that the issue will never reoccur. A reasonable level of confidence is more than sufficient for most projects, most of the time.
As an example of an environment where automated tests are critical: I read a post by someone complaining that when they worked at Oracle the automated test suite ran through hundreds of thousands of individual tests and took hours despite using a huge farm of servers.
That guy was wrong! I would absolutely test something like a commercial RDBMS to death. Those tests are not optional. Similarly, if I was developing a file system such as ZFS, I would also test the heck out of it. Famously, Sun had a test lab where robots would physically pull drives out of servers!
But would I test a typical web form with automated tests? No. It's boring CRUD code. It's going to work, because I use parametrised queries using strongly-typed ORM to a CotS database platform using "boring" code paths that are well known to work. If the DB schema changes, the build will fail.
I'm more interested in testing if the form UX layout "looks pretty" and has a "nice layout to help the workflow." Ensuring those will also exercise the boring parts of the back end code anyway as I click through the form a bunch of times.
Once the code has been established as working, and strong typing is used end-to-end, why bother testing it over and over?
You're making a set of assumptions when you write a piece of code, right? All the steps between when you actually submit a form and when the data is saved in your database are assumptions that you're implicitly relying on in order for your application to function. Individually, each of those components can work just fine, but no matter how "boring" they are, your application can still be broken.
For instance:
> a distinct sort was case-insensitive but should have been case-sensitive. It was dropping a few hundred items out of hundreds of millions.
You don't know that it's _going_ to work until you've _tested_ that it works. Right now, it sounds like your testing is almost exclusively manual. That's a valid approach, but it doesn't scale very far (neither in terms of application complexity nor team size and composition nor project age).
> Once the code has been established as working, and strong typing is used end-to-end, why bother testing it over and over?
You've established that it's working as expected _now_. What happens if somebody submits incomplete or malformed or outright malicious data? What happens if somebody opens two copies of your form, fills them both out, and then submits each of them in sequence? Which one should take precedence and does the notion of submitting this particular form more than once even make sense? What happens if your database isn't reachable but you have data that you need to save? What happens if another developer on your team updates the ORM and that includes a change to how data is sanitized on its way into the database that breaks some implicit assumption about how your data is going to be stored?
I hope you have some way to catch that stuff in code review. Syntactic rigor and being a bottleneck in the code review process is only going to get you so far.
Personally, I don't want to be figuring any of that stuff out at 3am when the pager is going off. I want to be confident that the team has solved _and documented_ all of the potential problems that they can think of (in the form of tests).