This is just silly, especially given you mention freezing releases and having an entire QA team go over the product. It would be much faster to have no QA team, no freezing of releases and instead have a robust test infrastructure.
It’s fine that this set of rules works for your company, you like them and are familiar with them, but that’s all this list is.
On the FE, we noticed:
* Most of the bugs were in UX/UI that are very hard to capture in a test suite.
* CSS/UI bugs are better handled with a good UI library. We have an _amazing_ UI dev that's built a really stable UI library for the team.
* It took forever to write FE tests.
* It took even longer to run FE tests.
* Even well-written FE tests miss a lot of subtle issues.
* We rarely had "random" bugs. UI bugs were almost always in the code we changed. Easily detect with quick click testing.
What _does_ work is front end integration tests with cypress/selenium. You just write some basic tests that click through the flows and see you got to the end. This catches issues all the time where the api response or something has changed unexpectedly and the UI gets stuck half way on an error.
It’s also super simple, it’s 10 likes of looking up elements by ID and running .click(). They hardly ever break from normal UI changes and prove that the app is mostly working as intended.
Switching to TypeScript is also a big one. It’s almost like half a test system given to you for free.
I would also state that "10 likes of looking up elements by ID and running .click()" is a big part of why e2e tests are flaky and hard to maintain. Good e2e tests require good engineering.
We've found that Typescript and a bit of boilerplate has almost entirely eliminated these types of concerns for us.
The primary goal is obviously to produce useful and targeted feedback as early as possible, but more strategically it elevates the status of QA, making it a gateway job to developer, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and a reframing helps to flatten the pecking order and improve access to the talent pool.
Before we all went remote work, I'd always recommend adjacent seating for quality folks, to exploit the natural proximity bias. Alas, I'm still not sure what that looks like today.
But really, I can see it. Firstly, a lot of engineers suck are writing tests. White box unit tests with interfaces and mocking everywhere tightly couple code blocks together. Integration tests are nice but take a lot of effort to get right while also keeping them performant. When (not if, remember we're talking about an early stage company) the codebase and/or the database schema changes dramatically it will often break the tests which is just another hangup.
In my pretty long experience it is a product issue, not tests'. And that is a kind of issue which sets the tone for that product development for years ahead.
>the codebase and/or the database schema changes dramatically it will often break the tests
That shouldn't break integration tests [much].
At that point most products are basic enough for developers to test the product themselves.
And if the codebase is undergoing lots of churn and heavy refactoring then tests unquestionably will slow you down.
Maybe it is better to only begin automated tests when you reach product-market-fit.
Tests force you to write better code. Code that is easier to change. Code that is more robust.
You don't need 100% coverage or perfect tests to gain this benefit.
A good test suite enables you to make large refractors safely and quickly.
The most scary refractors I've ever had to do are ones that involve large changes to code that has no tests.
Often those refractors that allow company level pivots require adding huge amounts of tests to a legacy code base before I can even get started.
Adding tests to code that has none is the worst. The code is coupled all over the shop and requires 100 times more mocking and stubbing to get under tests when you add this stuff retroactively.
Tests speed up development velocity and improve your codebase.
Just don't get stupid about it and you won't negatively impact startup performance.
I'm not suggesting middle layer testing is not valuable, shouldn't be done, or to never use mocks, etc., it is just much more difficult and more creativity is required to add value in this area. For example, in some situations middle layer testing requires the creation of various kinds of utilities and frameworks in order to find a reasonable benefit vs effort sweet spot. Dogmatic approaches (such as mocking all dependencies in all situations) feel like they are productive at first but can end up being costly as each mock is an alternate implementation of an interface and doesn't necessary capture the behaviour of the concrete implementation.
I don't strive for 100% coverage and I hardly ever do TDD unless I'm working on something hard to solve.
Tests give me confidence and results in less time repeating the feedback loop. I might have a slower "coding speed" but I have fewer regressions during refactoring. Less bugs to fix while iterating. And less back and forth with stakeholders to ship something. Overall I'm way faster with tests than without.
1. Feel free to write tests that help you develop your thought or solution. Not all of those should be checked in
2. Ask yourself: “What test do you need to write in order to protect this code from your future self next year when you have forgotten everything about this?”
People who don't test don't test because they don't know how. They are hacks.
Also hopefully interesting, the opposite of “don” is “doff”.
Or maybe they're just doing a different trade off. Do you unit test all of your shell scripts and jupyter notebooks?
If not, why is that? Is it that some code is either obvious enough or doesn't warrant having tests for it?
There's a spectrum of testing, from zero tests (which is fine for throwaway code), to testing a few key components and having end to end tests for the rest, to having 100% branch coverage. One should write enough tests that they're confident about the quality of the software that they're shipping, and no more, in my opinion.
* BE tests are easy to write, catch lots of important bugs, and generally speed up development.
* FE tests are painful to write, rarely catch meaningful issues, and don't really save any effort - devs still need to click through a UI to make it's correct.
* E2E tests were even more painful to write. Likely caught more than FE tests, but not enough to justify their even more insane overhead.
----
Typescript does wonders for FE test coverage. The bugs we do run into tend to be silly oversights (empty arrays, FE <> BE type mismatches, etc). Annoying, but quick fixes. Not worth a 30% overhead on every feature to save 5% occassionally.
In a similar manner, putting legacy code under test tends to remove a bunch of bugs; not because the tests find them, but because when I refactor the code to be testable, the bugs in the code become much more obvious.
I have nearly 100% unit test coverage for a couple of the base libraries I've created, but yeah, writing tests was slowing me down significantly for the rest.
Once I have something where chunks of the application aren't being scrapped every other week, I'll start adding unit tests to it too.
Something that's missing from this list that I find extremely useful:
An unstable environment with actual users doing real things with it. They could be internal, they could be advisors, they could be customers who are also investors or are willing to be testers for a discount.
That works really well if your code is designed to fail loudly (validations, assertions, etc...) and you have a culture of fixing issues immediately.
I think it is silly to use this one sentence without a context and start arguing. The whole paragraph in the article makes sense, gives context and advice is reasonable.
It also requires considerable skill to build and maintain, and not every team has that...
...in which case you might have a less good and useful test suite.
the latter is the only hope, but practically no one is willing to commit to it
now when I talk to people about making their product be something that development can be done on again - I try to figure out how serious they are. almost never.
It might be just me and a terrible string of bad luck bad I've seen so many projects that were kickstarted preemptively due to company politics or just management wanting to secure additional funding.
By the time (usually between 3 to 9 months) business stakeholders came to an agreement on what they actually wanted the team developed so much technical debt that it's virtually impossible to untangle from all bad decisions for the rest of the development cycle.
That's why I'm a huge advocate of starting simple, with a monolithic approach, to tread the water and have enough time to figure out the true objectives of the project first.