Unit Testing: Is There Really Any Debate Any Longer?
drdobbs.com
drdobbs.com
Instead, for complex or algorithmic code we're using generative testing, barraging the code with random inputs instead of a few cherry-picked test cases.
Then, we're structuring our systems and operations in such a way that we can spin up test instances of the entire system and run full, end-to-end integration tests at various levels of load.
We haven't figured it all out yet, but the results are promising compared to unit tests, especially tests which contain a lot of mocks and stubs.
I usually code in a terminal running screen with many shells going, one of which is running ipython so I can interactively import and test snippets, classes, or small functions that I'm writing in a larger module. It works great.
However - those tests disappear! Once a recent project of mine was handed off to a team for maintenance and more features, I found that, in particular places, where the code was subtle, it was subject to breaking by others who were just trying to fix something unrelated. This was when I, personally, realized the value of unit testing.
Yes, its slower and takes more time up front, but if your code may someday be worked on by others, it protects the sections of your code that have test coverage against breaking - which over a longer time scale can save time.
Tests definitely need to be repeatable.
Actually, it's often a sort of guided random testing. For example, if the algorithm has a straightforward linear regime with an exact solution, and a nonlinear regime that has to be solved iteratively, the unit test will target, say, 2% of inputs at the linear regime, 1% at the breakpoint, and the rest at the nonlinear regime.
Mockups and stubs are another matter. My algorithmic code is numerical, so stubs are rarely an issue. If an algorithm needs, say, a matrix solver, you pretty much just have to use a real (unit-tested) matrix solver in there.
There was a post on HN a short while back that showed why - in essence your edge cases sit at the edge of the normal distribution ergo they will rarely be hit by the unit tests.
This leads to you doing more work, as you need to log what data caused the test to fail. It also gives you a false sense of security, since the tests aren't really doing anything 99% of the time.
The problem is, writing testable code requires more thought up front not just about individual objects and methods, but also the overall architecture of your project. If your code is highly tied to an ORM or framework, then it's going to suck to test.
Also, not every language is equally easy to test. For example, Ruby has an obsessive culture around testing, so they have a lot of fantastic testing tools and the language itself makes things like mocking pretty easy. PHP by comparison doesn't value testing as much as Ruby does. The testing tools are worse and there are fewer of them. Also, due to the lack of monkey patching, you have to write your PHP code with dependency injection and testability in mind from day one for it to be highly testable.
In short, testing is a useful thing, but most code isn't written to support it and many languages and frameworks and patterns encourage you to write unpleasant to test code. If code is hard to test, programmers won't write tests unless their job requires it.
"Does this thing roughly do what I think it does?" is a secondary goal for me.
Very often, optimizing certain kinds of code for unit testing makes it significantly more complicated, with none of that complexity helping you in the actual program. This is especially true for "end user" code, such as controllers.
Unit test, yes. System integration test, no.
I'm not trying to imply that integration tests that hit the DB are always a bad thing, just that slow tests are another reason people don't like or use tests.
If I'm cranking out a prototype to show a customer then I don't want to spend precious time writing tests for code that stands a good chance of being thrown away at the start of the next iteration.
Tests often do need to be written, but the best time to do this seems like an unsolved problem to me.
I'd also like to throw it out there that testing can be especially helpful when writing code against known problems. Some benefits are:
- Since you already know the problem domain, your test writing will be extremely quick and effortless, so the benefit of having that test in the first place far outweighs the time/effort invested in writing them.
- In dynamic languages especially, typos are what kills you in this situation. Testing will quickly catch 'em as you program.
- Finally, testing will force you to think more about the edge cases, since implementation is a straight-forward process. Thus, you'll spend less time later doing QA and have an overall more polished feature.
There is no need to write tests for fast moving objects and for this kind of work TDD is a waste of time in my opinion. But for core infrastructure this makes sense.
But driving blind all the time is not a good idea on the long run.
The central point I am trying to make is that it seems wasteful to spend time writing tests for code that may soon be deleted.
After a few successful demos you're given an tight deadline to "finish things", so you have no choice but to polish up the prototype and ship it.
The moral: code that was written to be thrown away often isn't. You can still write code quickly, and take shortcuts (such as skipping tests), but recognize that you'll probably be working on that code for a very long time and may regret your decisions later.
I find myself wondering how many people complaining about unit tests work on leaf-node sorts of programming projects, because for me, many of the tasks I routinely take on in my environment where about a dozen projects are building on the code I am writing mean that the question isn't whether unit tests are "better" or "worse". It's a question of whether the task is possible or not.
This can be very expensive and take developers and QA engineers extra time when they try and figure out what's wrong... "Is the test failing because the test is out of date? Or is our program causing a bug?" I know for a fact HP deals with this issue constantly. Not saying unit testing is bad (I think it's usually good) but I don't think it's always good.
IMO, when you change code (extensively), you'd still have to invariably answer those questions. But in the absence of tests, it becomes much harder to find the answers.
(for the sake of this discussion I'll avoid subjective issues such as improved design)
1) Faster feedback cycle - if you're developing new code you can run a single test and check inputs/outputs very quickly. If you're testing by refreshing a page, or shudder, restarting your java server, you're going to have to wait far longer to find out if your code changes work as expected.
2) Reduced risk when making changes to the code base - the risk increases the closer you get to your data store as the amount of dependent code increases. The only way to reduce risk without unit tests in place is to not change any code i.e. you have to keep building on top (see the open/closed principle). This inevitably means that your code is going to introduce duplication as you can't refactor it out where appropriate. The other option of course is that you just risk it and hope to god that your manual QA testing will catch any problems before your users do.
It's also worth noting that the cost of adding unit tests isn't fixed, it's dependent on
a) experience
b) the tooling available for your language
From personal experience I find adding unit tests in ruby takes very little time. This means that benefit 1 in particular kicks in far sooner. Currently I feel it's faster for me to write a piece of code with a unit test than without.
I have friends in the medical industry, their tests must achieve 100% code coverage on multiple coverage metrics, and code coverage is only an afterthought to even more rigorous test requirements. Even they were pointing and laughing at the TDD ideologues.
Unit testing is beneficial, if you do the right amount of it and in the right places at the right time. You might have a totally sensible test plan. But like it or not, for the time being, plenty of people are unwilling to listen to you because some other assholes already burned them on the idea of unit testing.
Its going to take years and years to shake loose all the cobwebs in our industry caused by half-cocked advocacy. Then we can maybe get to the sensible discussion about how to analyze a project and develop a sensible test strategy that fits the project's needs.
2. Ideally, you should only test API not API's implementation but most of the mock framework will tie your test code directly against your API implementation. Once your API implementation changed, your test code fails, you have to rewrite test code, which is waste of time.
Someone may argue that it because you write bad test code; Talking is easier than doing;
So, usually, I prefer to doing the functional integration test than the unit test, which is productive for me. (I know people will throw out thousands of reasons about the benefit of unit test and "correct" way of writing unit test.)
Anyway, I will avoid unit test as possible as I can and ONLY do my designed integration test, which works for me.
3. And writing unit test for code coverage is stupid; Some manager even brag about the unit test coverage, which is totally a false confidence about the quality of the code underline; This is even stupid;
4. Writing unit test for those requirement-always-change area is waste of time;
5. TDD is a BIGGEST JOKE, which weights too much on writing auto testable code over writing working functional productive code;
@Test
testAddUser() {assertNotNull(new User("columbo").save())}
The biggest problem I have with unit tests is they tend to be
a) just testing frameworks (crud) and not much else
b) so heavily built on mocking that changing them is like redesigning a wool sweater. These tests tend to examine how the code was written, not necessarily what the expected output is (oh shoot, I need to mock up five more objects to make this test pass... hey look at that, this test doesn't even check what should be returned. Nice)
c) there is no context into or out of the unit test. The assertions are vague, it isn't clear how I got there, it isn't clear where I'm going (testXMLOutWithInProgressUsers ... what is the xml for, who are the in progress users... what is next in this chain of events)
d) the things that REALLY need unit tests don't have them because they are too complex to write (ah, that whole system that runs a nightly bash script, takes in a stack of csv files from the client and converts it in our database... yeah, that's not tested).
With all that said, tests are invaluable, but they need to actually, you know, TEST something.
I like nodejs in that respect. It takes me seconds to fire up the webserver and test the rest api. I don't mock objects, I pass values directly through the api and check the expected results. I like this because it shows the next user what the api actually looks like and I'm testing the full stack (In my experience controllers are notoriously undertested, mainly because you need the full stack running). Since the application is really just an api this means that I'm able to test everything equally, from batch processes to CRUD.
API testing has that wonderful property, its appearance is the whole deal, if it looks right then its working.
Anything but app code isn't so easily tested. When frameworks are written to support testing, they often are sub-useful - crippled versions of real programming languages etc.
Anecdote: I recall a QA guy asking me how to test our OS's file system. I said, make every filespec-based call with a good spec, a non-existant spec, an illegal spec, a null spec; log every result. After each release, diff the logs and explain every deviation.
He said "That's too hard, I'll just shotgun it." and left. Why ? Because his test-scripting framework wouldn't allow anything as sophisticated as lambdas etc.
So I spent half an hour writing a table and a loop, and sent it to him. He used it for the next 5 years to test every release.
My moral: do an exhaustive test when possible. The computer is fast, it doesn't feel any pain, and you will find more fencepost and boundary issues than you ever thought existed.
Granted, this is one of the reasons why I'm recently thinking about whether I've got a slight case of ADHD, but I think a lot of it just stems from the fact that I like to brainstorm code and afterwards use something like tests or just good documentation to reason whether that was the most proper approach.
Never mind that things like cucumber make me cringe big time.
That.
It takes all the fun out of using a dynamic language. You might as well go back to a static typing and write LESS tests for obvious things...
I try to find a compromise solution. I pretty much only write one unit test for a given module to start with, the nominal case. This is important, because it forces you to think a little bit about how you actually want your code to work, which I find to be a rather healthy thing to do.
Once that is done, a little later I will no doubt find a bug. When I'm trying to debug, and I start wondering whether a block of code is doing what I think it is doing, I write a test to verify. This does one of two things - it shows up the bug, which I can then correct, and verify that I have corrected, or it allows me to remove the tested code as a possible cause for the bug, allowing me to concentrate on other hypotheses for the cause of the bug, speeding up the debug process.
By the time I get to the end of the project, I generally have a pretty comprehensive test suite, without ever once feeling like it was a chore.
Granted, that's mostly for starting out. In an existing code base, especially a larger one, different rules apply. I don't believe in methodology panaceas.
But, when it comes to something I expect someone else to use, there is no question that I will start with a test.
In the big mean real world frequently you're shipping things in such narrow deadlines that if you say "Hey boss, I'm going to take some time to write some wicked cool unit tests, it'll make my life easier in future because we'll know this code works!" then Boss' head will pop as it's a non-tangible improvement.
Developer focused companies are great as they'll let you get away with this, but the large portion aren't.
I usually prefer "automatized system testing" where fewer "high level" tests involve a lot of code, preferably in a test environment that is close the actual production environment.
Of course, when a test fails, some more analysis is required to determine what went wrong. But it's usually related to recently modified code, hence easy to spot.
That way I get the benefits of automatized tests without the burden of tons of "micro" tests at the class/method level.
unit testing give you the false confidence that your stuff is working, it has little to no bearing on stuff actually working when pushed to production (unless you have 100% coverage and actually thought of and defended againt every possible use scenario before hand)
Yes there is a maintenance cost, but I find most people's objection to testing is based on poor testing skills. Once you get good at testing with a decent framework, you find that it's often about the same time to write formal tests as it is to do the manual testing which we all know is necessary to have any certainty that code is working.
The reason is that automated tests costs 2-3x as much time upfront. The assumption is then you will recoup that investment in the future. The problem is that automated testing doesn't cover everything so you have to augment it with human testers. Which then begs the question are the human testers being effectively utilised.
And for those of that do develop have a think about all the time spent on buggy web UI testing, screenshot comparison testing, tests being flakey due to software choices etc.
For some things like libraries and such, there are absolutely needed, but unit testing everything and especially TDD are stupid.
To be frank, they mostly sound like some Java EE-like monstrosity that for some bizarro reason caught on with the hip crowd too. I'd put them out there with FactoryOfFactory and XML documentation...
It's definitely some solution for testing. But when you're writing a webservice, you should think twice before assuming that anything a "UNIX programming god" says is a good advice for you.