Software Testing Anti-patterns
blog.codepipes.com
blog.codepipes.com
How does this myth continue to persist?
Writing the test first has nothing to do with knowing the final API. When I write tests first I am looking for an API. The tests help guide me towards a nice API.
I personally find TDD in this manner works best when taking a bottom-up approach with your units. I start with the lowest-level, simple operations. I then build in layers on top of the prior layers. Eventually you end up with a high-level API that starts to feel right and is already verified by the lower-level unit tests.
Although this style of development has become less prevalent in my work with Haskell as I can rely on the type system to guarantee assertions I used to have to test for. This tends to make a top-down approach more amenable where I can start with a higher-level API and fill-in the lower-level details as I go.
> How does this myth continue to persist?
Yes indeed. TDD helps you find a nice API.
You can TDD top down by writing an end to end test, and recursing TDD down each level until you get to a single unit, repeat until all inception layers of the tests pass.
You can TDD bottom up, by starting with components you'll know you'll need, working out some APIs at their level, then doing the same to piece together larger components.
Ultimately you're in control of how your API looks, and this has nothing to do with TDD itself - if you get caught up exposing a bunch of inner workings and configurations, then you're simply not designing well. It has nothing to do with TDD.
Likewise, if you get caught short with an API that's too simple, well, that has little to do with TDD either.
I merely wished to support agentultra's claim that
> When I write tests first I am looking for an API. The tests help guide me towards a nice API.
and to counter the article author's claim that
> Writing tests before the implementation code implies that you are certain about your final API, which may or may not be the case.
I did not mean my comment to be interpreted as "using TDD will guarantee you find an nice API". If you prefer, please interpret my comment as "TDD (under certain circumstances and when used in an appropriate way) can help you find a nice API".
I was discussing this with a coleague this week: starting by the lower level details vs starting from the more abstract/whole api. I prefer to start by writing a final API candidate and its integration tests and only write/derive specific lower level components/unit tests as they become required to advance on the integration test. My criticism on starting with the bottom-up is that you may end up leaking implementation details in your tests and API because you already defined how the low level components work. I have even seen cases where the developer end up making the public API more complex than necessary due to the nature of the low level components he has written. Food for thought!
That said, I don't think TDD is a good way to figure out your API. It has its value for sure, but not if you take a purist or dogmatic attitude towards it. In any case, it seems to assume that more tests are always better (hence you start every new piece of code with a corresponding test), when I'd argue that you want a smaller suite of focussed and precise tests.
There are plenty of ways to do it and one way that has worked well for me is in practicing TDD. So it is indeed one good way of doing it.
There are also people who are inexperienced with TDD or who misunderstand it and implement it poorly. There are people who are just not ready to design APIs yet who are forced to learn on the job. None of those things invalidate what I said.
If TDD doesn't work for you then you'll have to find what does and I hope that you test your software in a sufficient manner before deploying it.
> In any case, it seems to assume that more tests are always better
I think you're projecting your own opinions into the matter. There's nothing about TDD that prescribes what kind of tests one should write or how many. You could write acceptance tests and work from there if that suits you better. It's test driven development. The operative word is what we should focus on: testing is a form of specification and we should be clear about what we're building and verify that it works as intended.
It comes from an opposition to other forms of verification that used to be popular: black box/white box testing where the specifications were written long before the software development phase began and when the testing happened long after the software was developed.
That's where it's most useful: as a light-weight, programmer-oriented, executable specification language.
Unless you're talking about the default state of failing tests when no implementation exists, I'm a bit confused as to how TDD helps with interface design.
Over the years, I've learned to test "what a thing is supposed to do, not how it does it". Usually this would mean to write high level tests or at least to draft up what using the API might end up looking like (be it a REST API or just a library).
This approach comes with the benefit that you can focus on designing a nice to use and modular API without worrying on how to solve it from the start. And it tends to produce designs with less leaky abstractions.
Of course YMMV.
I think the latter gives an inflated sense of coverage (“But we’re 95% covered!”) but makes the tests far more brittle. What if you update a method and the tests pass but now a chunk of the API that references that method is broken, but you only happened to run a test for the method you changed?
I like to think I’m taking a more holistic view but I could also be deluding myself. =)
I should have said “you’ve only written a test” rather than only having run a test.
The problem is that in many cases management tries to push prototypes into production as quickly as possible and you might get stuck with the mistakes. Forever in the worst case.
Yeah, it means for people who don't have the time to rewrite the code and the tests multiple times.
The greatest and most poorly communicated benefit of TDD right there.
People who think TDD creates a good API probably have good instincts and a problem where an acceptable API is easy to create. When you have a complex problem that will have hundreds of users (thus it demands not just an okay API but the best) you need to think about the API first. Otherwise you can design yourself into a corner where to fix the API you have to change all your tests.
Once you have the API design in place TDD is a great tool for refining it. You need real working code using an API to sake out all the details and TDD is a great way to figure what you missed. You need a direction in mind first.
Well, what is good? You suggest such an API has to solve a complex problem with hundreds of users. I should think that I've written a few modules with just such an API using TDD methods. So what are we really saying here with this definition?
Your elaboration goes back to the myth I would like to see dispelled as my experience, and that of many others, has demonstrated that it's a bit of a red herring.
When I start a module I don't know what the API should be or what invariant is important or anything. All that test driven development asks is that I first make an assertion about my problem and try to prove that it's true (or not as the case may be).
This leads me to good API designs because the tests invariably contain the assertions about my invariant, they document the API at a high level, and they demonstrate that my implementation is correct as to those assertions I made. If I am thoughtful in this process I know that some assertions contain a quantification over a universe of inputs so I use property based tests to drive my specification. Other times a handful of examples is good enough to demonstrate correctness and so I simply test the units with them. And the API is entirely under my control this whole time. The tests are guiding me to a good API that does exactly what I say it will do by way of executable specifications that cannot be wrong about my implementation (or else the test would fail).
There's nothing in the above that says I cannot take a top-down approach and design my module's interfaces first. Nothing.
Personally I prefer the bottom-up approach for many of the reasons I explained in my OP. I like to define things in small units that I can compose over with abstractions that let me achieve more interesting results. Testing keeps me honest as to the true requirements so that I don't go off into the woods.
What you are missing is that TDD gets you to AN answer, but it gives you no information that you got the best answer. TDD does make some bad design choices hard or obvious; but there are other times where that there was a better choice isn't obvious.
Bottom up and top down are very different design considerations. You can get bad APIs in either one. You can do TDD with either.
I'm a fan of TDD and use it. However I'm not blind to the limits.
When working on protocol specifications and proving the correctness of a certain invariant or property of the system I find TDD to be lacking and tend to turn to formal methods for results.
However at the module/interface level of an individual software library or component TDD is a very good tool that helps guide me towards a good design. As do other tools like a sound type system. When they work in concert the system almost writes itself -- I just have to come up with the queries, invariants, and assertions.
As much as I'd love a very cleanly implemented API, in the end many more devs will be using the API, so the use of it has priority over an API that is easier to implement.
- Company starts out, lots of change and RFCs, testing at a shallow depth (high-level tests wherever possible) helps a ton
- [Goldilocks zone] Company and team settles down, predictable workflow, "not changing the world" anymore, unit tests help
- Project goes into legacy, needs to be split/ merged/ refactored/ rearchitected/, again unit tests don't help. Back to shallow tests and stranglers
Only very small part of programming today is doing hardcode datascience and algorithms. Rest are all just plumbing from A to B to C. I always argued in teams to focus on higher level tests, but then, its easy to get shouted down because of "TDD". Sadly, very few people seem to focus on objective rather than process.
In my experience, your inclinations are true, but it actually takes more discipline and buy in from leadership for it to work. If broken tests are lower priority than feature work, your test suite ends up rotting, getting ignored, then abandoned.
At least with unit tests, individual contributors can maintain pockets and layers of verified-working code.
I'm not arguing against any kind of test. But you do need the people handing out raises, planning sprints, scheduling deadlines, etc. to be on board for end-to-end testing to work. It also helps a lot to introduce those early in the system's life, since the team culture develops with the tests already being there.
If your product consists of somewhat reusable libraries that you compose together, and extend when needed, having strong unit tests for the low level functionality helps immensely when it comes time to refactor. You know, stuff like "when given int(2) and int(3) as arguments, add(int, int) returns int(5)".
Likewise those high level API tests that ensure the system-as-a-whole is still doing approximately what it's supposed to, regardless of what mess of wires and small aliens is inside the black box are super useful for moving forward as both proof of functionality and guard against regression.
It's the stuff in the middle that's icky IMO. Integration tests that are testing your glue that, especially these days with clouds, containers, microservices etc, can be quite far removed from any codebases you might maintain. I don't have a good solution for it, but I'm searchin'...
https://blog.kentcdodds.com/write-tests-not-too-many-mostly-...
This is advice I generally follow - most of the time, I find I get the most value from integration tests.
This is a great essay but I'd say it's the ideal and I would challenge Anti-Pattern 2 in many cases. The time, money and lack of resource crunch is always present. In the video game industry where I've been responsible for automation infrastructure around testing I would say there's the added difficulty where projects can have a short lifespan or be cancelled entirely quite quickly.
Integration tests are highly valuable for being a catch-all for issues. Having only integration tests without unit tests is dramatically less work to both setup and maintain. They are most 'bang for buck' because they can catch so many issues. The points mentioned, that they are complex, slow, and harder to debug than unit tests are all true. However there are some ways to counter these problems. In the section marked 'integration test results' he writes that it's hard to know where to start debugging and that a unit test would have isolated the issue to a smaller scope. While that's true, I don't believe that's a good reason to write unit tests because this issue can be resolved by having better logging and reporting. The integration tests shouldn't just report a fail and nothing else, report the specific reason for failure or at least save off the log so that this information is easily extractable. Also useful is to save screen captures, if there's a GUI, and call stack and crash dump data, if it's a crash.
With integration tests and either few or no unit tests and great reporting you can get enormous value from automated testing. Most real world environments will demand this sort of high ROI for less unless it's projected to be a very long running project or has massive funding, like a government program or grant.
Automated testing saves companies so much cost and helps deliver higher quality software faster, especially with larger number of devs checking in constantly with continuous integration. You're not doing it wrong if you break some of the rules in this essay, you may be required to be more programmatic. There are some great points in this though.
In the small places I have worked, it been acceptable to have a few bugs, and the applications were completely or partially used by internal users, so maybe 40 -50 people who can put up with a bug here or there and report issues personally.
Unit tests are less expensive to set up on a per test basis but the code is less reusable and the ongoing maintenance tends to be higher.
1. Integration tests let you know that something is wrong but they are mostly useless for locating bug in larger code base. They do not help you much when you write code.
2. Unit tests force you to think about code structure. Code not covered by unit tests is usually much worse in terms of readability and maintainability. (It is hard to write unit tests for huge functions or for spaghetti code but integration tests are not affected by this.)
This is my biggest issue with TDD proponents. Like... you don't need TDD to learn to code properly!
I think you answered the question yourself. Speed is also shown in the table that defines the types of tests.
You can name the categories foo tests and bar tests if you wish.
Anti-pattern 1 - is companies that have only the "fast suite" you are talking about and anti-pattern 2 is companies that have only the "slower suite".
With this you still have fast and slow perhaps, but everybody can run all but the most extreme scaling tests.
The performance of such a test is still a problem.
it('sends an email to the user', ...)The speed at which they run is their side effect. It’s possible to write slow running unit tests
In OO languages, I personally prefer unit tests with all dependancies stubbed or mocked out to verify the interface of that specific unit or component.
In test code, I am a lot more tolerant of straight-line repetition in the name of simplicity, in a manner that would be worth abstracting were it in non-test-code.
Just kidding.
Sometimes I do something stupid like put an extra comma somewhere without noticing, or update my db driver library without also updating my mock library. I think we've all had that moment where tests fail or it doesn't compile because you thought you copy-pasted some text into an email, but it got pasted into your editor window.
So when I see that literally every test has failed, I know it was some really dumb but likely simple mistake on my part. If 180/195 tests pass.... Shit.
I have seen however more projects out there that have code duplications in tests than projects that go overboard with DRY
This applies to all code, not just tests. Dry can turn some stupidly simple if somewhat repetitive code into a complicated mess.
For unit tests though, as I general rule I don't share any setup between tests but I use the test fixture setup method for common mocks to return default answers. Even that can get too hairy at times and it's better off creating a second test fixture.
I agree with "code duplication" and "copy-paste segments" (although I fail to see how they're two different things. It looks like example duplication to me?)
I don't agree with hardcoded variables. In tests, they're okay. Tests are not production code. In this case I side with Misko Hevery - see https://youtu.be/jVxmk-tVo7M?t=2m54s
Tests should hardcode values, and (ideally) not contain any logic, not even of the simplest sort. That's the point, that's how we can reliably confront them with production code without risking bugs that fall off off our radar because the same faulty logic leaked from production into tests. Tests should be kept all dumb, all naive, all hardcoded.
That's the thing which proves to be the most difficult to convince fellow developers about, as they're typically used to mechanically transferring over all the usual best practice from production code into tests.
"Ooh, this is so hardcoded" - "good". "This method name is so verbose" - "good; will we ever call this method from code?". "This could be private" - "it's a class with unit tests, how will this code possibly be called from some other class while it executes? Visibility modifiers are waste of space here". And so on
I used to think this was a good idea too, until I saw the real statistics on a project on this.
This project (50+ developer team) tracked all bugs, and also if they were regressions or not. Almost 0 regression occurred of bugs that were fixed before.
All testing needs to consider return on investment. The reality, at least for that project, was that testing time was best spent elsewhere.
if there is some brain fart, and its tracked down and fixed, the value of encoding that in a unit to be run until heat death. as you say, exactly zero.
but if there is a problem area that keeps throwing bugs like a metastatic tumor - say race conditions or memory footprint, or unicode handling. in that case the return on writing tests to try to shake out those areas is pretty high.
I think the real failure here is to try to substitute informed judgement calls and meaningful discussion with simple checklists of dos and donts.
I agree, but in that case you shouldn't test for that one specific case that failed, but try to make a test that handles similar issues, or tries catch bugs of the same kind.
Of course it depends how far you can push this.
I personally believe that this advice is one of the most important one actually for a very specific reason: it leaves a very bad impression to client when a bug keep happening again and again after being fixed. Unfortunately I've lived this experience when I was just starting out in my career on a similar project (around 45-50 developers) with basically no tests at all. It wasn't fun explaining to the client, even if they were internal to the company, that the bug we fixed last month had to be fixed again.
Only in the sense that:
- What's this?
- A device that keeps tigers away.
- But there are no tigers in Los Angeles!
- See how effective it is?
>The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine?
No, he means that they had code to test for the presence of fixed bugs, but nobody reintroduced said bugs and triggered that bug catching test code ever. So even if they didn't have the code, the end result would have been the same.
Perhaps the tests don't run on the devs laptop, but on an integration system (we have such a setup).
(And of course you can make test suites report failures centrally, whether they tests run on a laptop or not).
Integration tests were really heavy, and ran multiprocess on a server. So locally we would run the unit tests, and relevant integration tests. The rest was tested on the integration server, which automatically created bugreports when something failed.
Except that there are, and are kept from terrorizing the population by very effective devices: iron cages. (iirc there is at least one Sumatran tiger in the LA zoo)
Very effective. Especially if you consider that there weren't any ;). You can't get any better ROI of zero effort! ;)
Each project is different of course, and if you saw this in your project, than it would probably make sense to implement them.
But all I'm saying is that this shouldn't be a hard rule. Look at the project, look at the current issues, track your bugs, and draw your own conclusions.
It’s not quite the same though as adding a test to verify a bugfix.
Manually testing your bugs is still cheaper in time and effort than writing unit tests. If it was not, everyone would write unit tests.
Unit tests need to be able to fit into your codebase, sometimes contain bugs too, need to be maintained, slows down your release cycle, etc. I'm not claiming they are bad, I'm claiming it's a trade-off that you make in favor of quality, stability, etc.
Unit tests also need to be produced by expensive, probably scarce programmers. Sometimes it's better to hire 'cheap' testers that can script tests into your framework.
That's why I like to talk about return on investment. If you put in the extra effort of writing a test, you expect this to pay off in the long run.
Nothing is cheaper than not testing your bug after you fixed it. But the chance that a lot of time will be wasted afterwards because it wasn't actually fixed is huge. So basically you have to find the most optimal way to spend your effort.
And like I said previously, our statistics showed that writing unit tests on found bugs wasn't the optimal way to spend our programmers effort.
Automated testing should not be the end of the testing. It should be the beginning. Manual testing should be the last step, even if every manual test is immediately automated - there's always more to test.
Also, we should be sure to include combinatorial and fuzz testing in that pyramid as well, since skipping them leads to someone coming in with AFL and exploiting the hell out of your app.
I strongly disagree with this one. I think that unit tests only make sense when the project's code has really settled down (not likely to change in the future) and you want to lock it down to prevent new developers on the team from accidentally breaking things.
Unit tests severely slow down development. I've worked on projects where it takes 2 to 3 days to update a single property on a JSON object on a REST API endpoint because changing a single property means that you have to update a ton of unit tests. The cons of unit testing are:
- It locks down your code, so if your code is not structurally perfect (which it definitely is not for most of the project's life-cycle) then you will have to keep updating the tests as you write more code and more functions around.
- It encourages you to use certain patterns like dependency injection which might make sense for some (e.g. statically typed) programming languages but are unsuitable for other (e.g. dynamically typed) languages because they make it difficult to track down dependencies.
- It only makes sense for parts of the project that have strict reliability requirements and where any downtime/failure in that part of the code would result in some loss of business. It's important not to underestimate the maintenance cost of unit tests. More unit tests means much slower development (cuts productivity to half or sometimes even a quarter of what it was without tests for small teams), which means that you need to hire many times more developers to get the same productivity that you could get from a single developer. Sometimes it's OK if a part of the code breaks in non-critical parts of the system; especially if you have some kind of user-feedback system in place.
You just described anti-pattern 5. Did you read the full article?
>> Tests that need to be refactored all the time suffer from tight coupling with the main code.
What is the author proposing? To write unit tests that are only 'loosely coupled' to the code that they are testing? In my entire career, I've never seen a single unit test case that matches this description.
If it's loosely coupled with the internal code then by definition, it's called an integration test.
Anti-pattern number 5 is basically the author admitting that internal unit testing is a problem in terms of productivity but then they fail to offer an actual solution which doesn't contradict the rest of the article.
Sometimes your code needs refactoring, you need to change the fundamental structure of how some objects interact with each other and when that's the case, unit tests actually discourage you from making the necessary changes of pulling the whole class definition apart (thereby invalidating all the unit test cases for that class) and moving the code to smaller or more specialized classes.
That is not an argument. The fact that you have been doing something your entire career does not make it correct
>If it's loosely coupled with the internal code then by definition, it's called an integration test.
That is your own definition. The article defines an integration test right at the start. It is ok if you have your own definition but that does not mean that everybody has to agree with you.
>Anti-pattern number 5 is basically the author admitting that internal unit testing is a problem in terms of productivity but then they fail to offer an actual solution which doesn't contradict the rest of the article.
The article has an example and shows both the problem and the solution. The solution is to make your tests not look at internal implementation. What more could I do there?
>unit tests actually discourage you from making the necessary changes
you are just describing again what anti-pattern 5 says.
>What is the author proposing?
I am the author, so I know what I am proposing, that is for sure.
How do you balance this?
One technique i use is instead of splitting the test in unit/integration I try to find the most stable APIs in the codebase. So you don't make a complete end to end A->B->C->D nor individual A, B, C, D. Instead you divide it into smaller integrations such as one test for A->B->C and one for C->D, assuming the C interface is stable.
> That is not an argument. The fact that you have been doing something your entire career does not make it correct
I've worked for many different tech companies in my career (both startups and corporations) and the vast majority of these unit tests were not written by me.
Also I've worked on many open source projects. Same story.
Let me put this way. I am writing an article on how to keep your body healthy and provide a list of common mistakes.
Anti-pattern 5 is "you should stop smoking".
And your argument is "I have been into too many companies (startups and corporations) where people have been smoking all the time. So anti-pattern 5 is wrong."
Is this more clear?
Examples of what? Python? Ruby? Java? C++? You cannot please everybody. The article is written in a way that touches all developers. And judging by the feedback I got it has succeeded in this way.
>that basically amount to good tests are good and bad tests are bad
And the article helps people to understand which tests are good and which are bad in the first place. If you are already an expert on the subject then maybe you are the wrong audience.
>Who reads this and is any wiser as to how to write valuable tests?
If you think you can do better then by all means I am expecting your article on the subject
>What these people need is not another article talking in loose terms about the wonders and virtues of testing.
Please write the correct article then.
The original interpretation means a unit of behaviour. When you start thinking like that, intergration tests become less important. But still important.
The idea that a unit in unit testing was a class was a misinterpretation.
The first project I worked from after graduating had this problem when I came onboard, all the tests were slow as all hell and 80% of each test did the same thing (clearing the entire database and seeded in new test data, for every test). I felt like I made a mistake becoming a programmer because of how gruesome this anti-pattern is when you work with big Java applications. Then we hired a senior developer three months later who promptly started breaking up the tests after checking with the team if that was ok. The productivity of the entire team increased by many orders of magnitude!
Don't let problems like these disappoint you. There are companies that don't suffer from any of these anti-patterns
Focusing too much on fast feedback
Nope. One should get priorities straight. The #1 purpose of tests is safety. The "sleep well in the night" test. The "deploy with eyes closed" test.
Performance of the test suite, while "nice to have", is not the core objective. If push comes to shove, if it comes down to safety vs performance, safety should just win hands down as a principle.
Doing a bunch of mocks for speeding up 80% of unit tests? Great, but its a borrowed debt, and must be balanced out with 20% of higher level tests.
Just because you can run all the tests and all the optimisations on every keypress doesn't mean you shouldn't have them.
I started with Extreme Programming in 99 and grew and evolved through all the refinements to the process around testing. It's always been about providing safety. Fast has always been about not running silly tests that take too long
Fast Feedback is a core objective. It is not in competition with safety, safety is an integral part. Safety is the feedback you are expecting. We want to know about that safety fast.
A popular term that came around in the early 2000s was "Brain Engaged", meaning you needed always to be aware of why you were doing things and not following blind rules. Meaning you need to know the purpose of going fast.
The whole point is to go as fast as possible safely.
Some of the biggest challenges is how to get things quick while maintaining safety. Kind of makes no sense to have fast tests with no safety.
Now you mention mocks, and I have seen people mock in very strange ways that devalue tests.
I like the general guidance from Kent Beck "I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence"
Lots and lots of unit tests for the mathematical equations.
Some integration tests for the CSV reading and JSON writing
No UI tests because there is no UI."
As a developer, I feel that others should test my code. Is that reasonable?
Who says I don’t?
In my previous job this worked quite well.
I discovered those drawbacks fairly early, so quickly made it a habit to be super-strict and critical about pretty much every single line of code I write (both with regards to style, functionality, how it adheres to the standard practices etc).
What I also do consistently is review older code: if I have to add a function to existing code, it happens more often than not that I'll re-read and review the surrounding bits. Often to come up with better code. If time permits, else I take note and go back to it later.
In practice it turns out that if I haven't touched code for months, reviewing it is almost the same as reviewing someone else's code, i.e. starting with a clean slate. Which is also why I'll sometimes write something, commit it, but not yet merge it to production. Then a week or so later I'll come back and review every bit again.
I personally find that me a couple months down the track is pretty much another person anyway. In a sense you're not reviewing your own code, you're letting it be reviewed by future you. But if you're in a team with multiple developers, having someone else review your code would be more efficient.
I really like this idea, haven't heard it before but it makes complete sense, thank you.
As an example, we wanted to automate the test for browser-based OAuth connections to GitHub and authorizing an app there.
Here is the test that does so: https://github.com/coreinfrastructure/best-practices-badge/b...
Even for manual tests, the test spec should make part of the change request and be reviewed appropriately.
Cheers!
Frankly the only good source of such material is books (especially the initial chapters where they set the stage for everything)
If somebody else knows where these types of articles exist, I would also like to know where they are.
OOC, how long did this take you to write?
Most of my time was spent on thinking the examples and trying to structure the content. The actual writing was very easy when I knew in my mind what I wanted to say (as is always the case with technical writing)
You could do the same thing by "F5"-ing it, that is running the full app, but the problem with that it is slower and above all NOT repeatable later, without significant setup time.
Running the test, sets breakpoints and debug implementation by running the unit test, is by far the fastest way to develop, while also having atleast some assurance that the code you are writing is "correct"
We have all integration tests, and no unit tests. I and several others have pushed for unit tests, but with little success. Our full test suite takes 10 hours to run. We have split it up so we can test what we think is the portion we are modifying, but we're never 100% sure something in the rest of the suite isn't dependent on our changes.
I disagree a little with his complexity multiplying for his anitpattern about no unit tests. In theory, yes, you would multiply them. In practice, it is rarely the case you need to test all combinations - real code rarely looks like his diagram. In our experience with our project, the final number of branches we need to test is much closer to adding than to multiplying.
I very much agree with the "don't test internal implementation". If the primary reason your tests fail is because you refactored or made API changes, your test suite is not robust.
Am having to live with flaky tests right now. Horrible. Our team doesn't want to prioritize fixing them.
One antipattern he left out: Making unit tests a 1:1 match with code, and insisting that a unit test should not test more than one funciton. I know the community is split, but I am very much in the camp that "unit" should not be tied to a function. Don't make it that granular.
Basically, 95% of the methods depend on database [and it's current state].
How can I unit test this?
I've given up on unit tests. They wouldn't make any sense. E2E tests are helping us a lot but I had several attempts to have unit tests but they don't make any sense at all.
So test that with mocked data.
Unit tests should be able to run with no network connection, no other service, just your test file + data.
Otherwise at that point you’re doing integration testing, not unit tests.
Basically it gets an array of id's and calls the database and fetches those entities.
~10 lines of Javascript code.
40 lines of SQL. Which also depends on the state of the database.
So what do I gain by mocking the database? Making sure those 10 lines of javascript work fine? In a vacuum? They are the 10 easy lines anyways which are covered by my E2E test.
What matters is the SQL and how it behaves in different states of the database.
For your unit test you’d just mock the dB call and make sure that function called the “db” and returned the faux data. That’s it.
No one said unit tests had to be complex or test the whole stack. Quite the opposite actually.
That’s like... 2 minutes of test writing, tops?
I think that we really need to discuss unit testing as a group more. Specifically: what a unit test isn’t. It’s amazing how popular some misguided and ill-informed ideas about them are...
If your units of work are "retrieve data for a list of IDs" and "store this data in the DB" then there probably isn't a lot of unit testing to be done. You probably have some data cleansing or validation functions that you can unit test, and probably some domain-specific data transformation that you can unit test. But most of your testing should be integration testing because the important thing is that the piece that's reading the data and the piece that's writing the data work the same way. If you only have unit tests that set themselves up and tear themselves down then you'll never find a bug where, e.g., the columns get switched around on a write.
I also agree that "Paying excessive attention to test coverage" is not good. However, I completely disagree with much of its supporting text. If your test code coverage is only 20%, then by DEFINITION your tests are awful. That would mean that 80% of your code is completely untested. I agree that for many programs 100% code coverage is not worth the effort, because those last few percentages cost more than their benefit, but that doesn't mean that such low coverage makes sense. Most organizations I've worked with recommend at least 80% statement coverage, as a rule of thumb. I haven't seen any studies justifying this, but this essay doesn't cite anything to justify its claims either :-). You'd want much higher statement coverage, and also measure branch coverage, if software errors are serious (e.g., if someone could be physically harmed by an error). You should focus on creating good bang-for-buck tests first; code coverage is then a useful tool to help identify "code that isn't getting well-tested at all." It's also useful as a warning: 100% coverage may still be poorly tested, but low coverage (say less than 70%) means the program definitely has a terrible test suite.
This statement is misleading: "You can have a project with 100% code coverage that still has bugs and problems." That's true for ANY program, regardless of its testing regime, because any testing regime can only test an astronomically small fraction of the possible input space. A program that just adds 2 64-bit numbers has 2^128 possible inputs; real programs have more.
That is not always a bad thing. Maybe depending on the application this 80% is trivial and never breaks. It is explained in anti-pattern 4 that you should start with the critical code first.
>Most organizations I've worked with recommend at least 80% statement coverage, as a rule of thumb
This number is making MANY assumptions.
I would demand different code coverage from an application that runs on a nuclear reactor and from an application that is used as point of sale in a small pizza restaurant.
I think there's a deep issue that causes all the misunderstanding, the elephant in the room : the definition of a "unit". Words have a meaning in a certain context : if people don't mean the same thing when using the same word they're doomed to misunderstand each other forever. Just ask 5 different people what is a unit and you'll have at least 3 different definitions. The most common one is : in OOP, a unit is a class.
From my experience, a unit should be defined as a much higher abstraction level than that. A better definition would be : "a set of use cases that belong to the same module". In other words, unit tests should be we written in a language as close as possible to your domain language. Or : "test your use cases, not your classes". When you do that, you usually write tests for a few major classes that use all your other classes that are just implementation details. This leads to tests that are far more reliable and easy to maintain, because they have very low coupling to the rest of your application. Typically, this means testing classes at the very edge of your app, classes that directly communicate with the end-user, usually services or something like that.
Let's say you "unit test" a simple car with a steering module. It has all sorts of internal complex mechanism that IMO you generally don't need to unit test directly. What you need to know is if the business value is correctly delivered to the driver, ie :
- when he turns the steering wheel left, does the car turn left
- when he brakes, does the car stop
- when he presses the gas pedal, does the car accelerates
- etc
Under the hood there are dozens of other classes that perform actions that you don't really need to care about when you test. They will be tested indirectly anyway, because they're used by the high-level classes you do test. I think many people blindly try to unit test almost all the classes they write and it leads to code duplication all over the place and all sorts of other problems that make projects fail, people angry and think unit tests are bad.
I run up against this the most. The internal state often does not matter. What is important is the bevhavior. Refactoring internal state should not break a bunch of tests.
I think devs need to keep it simple, even it leads to more code, so that a wider range of tests can be written with the same abstractions.
None the less the article does a good job covering lots of the reasons writing tests will result in a bad outcome. Much like the reasons developers dislike certain architecture patterns, it's not the pattern that's at fault but the application of it by individuals and teams
But what if a whole load of the logics are in stored procedure? Surely those needs test too.
Of course, one can make an argument that depending too much on stored procedures is an anti pattern....
Works fine with Reader View, though.
The other alternative would be a loan approval application.
Remember that the definition of a developer is very broad nowadays. You have hackers working with C/Assembly on firmware, all the way to AI/ML with high level languages.
Do you have another suggestion for a good example?
Did you actually read the entire article? It specifically mentions this as a anti-pattern, having tests testing the wrong thing and you should figure out what's the most important thing to test and how. Read the section "Anti-Pattern 3 - Having the wrong kind of tests" and you'll see you actually agree with the article.
http://blog.codepipes.com/testing/software-testing-antipatte...
at first I took your comments about "the shape is not a pyramid" as there being something wrong with that and that the shape should be a pyramid.. reading a bit more carefully I now see that's the opposite of what you're saying...I blame everything on english being my second language O:-)