How deep are your unit tests?
stackoverflow.com
stackoverflow.com
The whole of GNU+Linux for example (or at least most of it)
On the other hand I've seen several projects claim "very good TDD coverage" and then crash and burned when put into production (usually posted to HN)
Real world, real usage testing is essential. TDD is good for keeping you on track and avoid regressions
It's also good for complicated (small) pieces of software that do something complex and is prone to having its behaviour adjusted with time (think: reports, data consolidation or analysis, calculations, etc)
If there's need for explicit tests, just use the code in the right way and see it exit cleanly. Then use the code in the wrong way and see it assert (first replacing the assert() with a throw/longjmp to catch failures without aborting).
Asserts also double as a concise context documentation. When an assert is triggered, it's typically easy to see what its condition means and what has gone wrong. So it's a win-win all around :)
The advantage with assertions as well, is if you leave them on in production, when an issue hits, you know about what it is straight away and can fix.
Many of our system errors are fixed and deployed before the user has finished raising the issue.
I'm not a fan of unit testing. I'm just doing it because it's a requirement (and we still have no QA).
Why I'm not a fan? Because I'm doing my own tests. And probably it has a bias all over it. I know when it will have a successful run and I expect where it will fail. This is not the scenario that I wanted. (I know this is not a good testing scenario)
In my opinion, a programmer must write a good code. And leave the testing to the QA.
I always write good code, but somehow that code turns to shit after six months, and I promise no one touched it. So, I guess I don't need the tests when I'm writing the code but I need them badly when I maintain it.
I still try to avoid errors, but it's pretty much nigh impossible.
It is written by Kent Beck. He pretty much invented the concept of test-driven development. He's probably the most well-known advocate of developers writing their own tests there is.
What his answer demonstrates (to me) is that the conclusions he and the other XP guys arrived at - to write code in tiny chunks, each one preceded by a tiny failing test - is based entirely in pragmatism. He does it because it works better than anything else he's come across, including leaving the tests up to a dedicated QA team.
Except that it hasn't been shown to work better. Quoting from http://third-bit.com/blog/archives/4529.html: "If you ask its advocates for evidence, they’ll tell you why it has to be true; if you press them, you’ll be given anecdotes, and if you press harder, people will be either puzzled or hostile."
There is very little research which shows that TDD is better than other approaches. Perhaps a bit, for some cases, but not enough to be certain about its general applicability. (Or has this changed in the last three years since I researched this topic?)
I hate writing code in "tiny chunks, each one preceded by a tiny failing test." I prefer to use a small set of burn tests until I have the main structure in place. Only then do I do the tiny tests - through the high-level API where possible (so I can refactor later) - and I use coverage analysis to make sure I'm not missing any obvious tests.
Coverage analysis is not part of TDD, which is a shame. (Technically that's because TDD is a design methodology, while coverage analysis is part of testing. While TDD generates tests, these are not a complete set of tests. For example, security analysis and scalability are outside the scope of TDD.)
Oh good, evidence against is just as preachy and unscientific as the evidence for.
http://evidencebasedse.com/?q=taxonomy/term/2
Research suggests that unit testing has a very strong correlation with correct software. Research suggests TDD has no correlation with correct software, except for the fact that TDDers tend to write 2x more tests than non TDDers.
You don't have any, do you? So obviously I can now promote it as the new universal methodology.
(Or more directly, you think the argument is that there is evidence for and evidence against TDD. My argument is that there isn't enough evidence to make a conclusion.)
Given that this is a discourse, have you read any of the discussions of the topic of TDD? For example, "Making Software" (e.g Oram and Wilson) has an excellent chapter summarizing the research results. If not, then it's a rather unbalanced discourse, no? You pointed to a list of papers, but didn't describe what you wanted to discuss from them.
For example, some of the research compares test-first with test-last. I'm a proponent of test-during, and don't think that test-first and test-last are the only two ways to compare things.
You would need to compare teams of equal skill level who were working on very similar software over quite a long period of time.
What would the benchmarks be? Number of defects in production code? Time to ship? Severity of defects? Time spent debugging?
Besides, that my point, isn't it? Promotors of TDD argue that their personal intuition suffices, despite having little evidence to back their claims.
[1]: http://www.haskell.org/haskellwiki/Introduction_to_QuickChec...
This may sound complicated and seem like overkill, but I've actually found it easier to use this style of tests for much of my code. Having randomly generated inputs can help find edge cases I did not even consider when writing the code in the first place--this addresses the bias issue you are worried about.
I also find that this style of test results in concise, easy to read tests. The usual "hello world" example uses the reverse function for lists. We want to ensure that for all lists, reversing it twice does nothing. The test would look like this:
prop_reverseTwice ls = reverse (reverse ls) == ls
While real invariants are often more complex, I find they are still usually easy to read right from the code for the test.
Anyhow: you should really try QuickCheck or something like it for your language of choice.
However, in other cases, the usage is less obvious. For instance, suppose that you are writing, say a URL parser. What do you want QuickCheck to generate and check? If you generate too broadly, the invariant check becomes a parser itself, if you generate too specifically you might introduce bias again.
Amusingly, I also wrote some tree diff code in Haskell recently. However, I did it rather poorly so I'm going to have to rewrite it soon (probably some time during winter break). I definitely agree that algorithms like this are particularly well-suited for QuickCheck.
Aside from the "not my problem" vibe that gives off, I strongly disagree. QA is helpful for covering weird user interactions and writing tests against bugs discovered in production, but you as the programmer should know what your code is supposed to do while you're writing it. Best write tests then rather than six months later when you've forgotten what it was for.
I see some people say that you should test each object in isolation, mocking out any dependencies. That seems wrong headed to me.
I prefer my unit tests to test everything all the way down.
The only thing I would mock is the file system, which is useful for testing file loading code.
They already have that, that's called integration testing. Unit testing is trying to test as small of a unit of code as possible - this helps you identify exactly where the error is. If all of your tests are integration tests and something fails, you have no idea which part of your stack the problem is in.
Obviously different projects require different kinds and amounts of test cases, but I prefer a mixture of both unit tests and integration tests (unit tests for small, complex blocks of logic, integration tests for most of the other stuff).
Not necessarily, because if you have a lot of integration tests then probably multiple ones will fail given a bug in a certain module, and the pattern of the failures might be all you need to locate the source of the problem.
One of the most successful projects I ever worked on, in robustness/quality terms, didn't really have any unit tests at all, but it had a comprehensive suite of end-to-end test cases that could be run automatically. Many of those didn't (and couldn't) have an absolute true/false outcome, either, but looking at the results they generated and applying various heuristics developed from experience, they were remarkably useful for similar reasons to unit tests.
Our solution was to start mocking as many models as we could, which really did improve test runtime.
(Slight edit: after reading the other replies, now I really understand what integration tests are for.)
User.active.in_group(x).not_replied.each { ... }
Unless there's a reason to mock something up (like simulating a fail condition from a DB or network call, or simulating other code that isn't finished yet) I prefer to just test everything as close as possible to what is actually running. The level of granularity, like the OP said, changes depending on the complexity and the critical nature of each part of the code.
Some people argue that if I make an error in function A and that causes a unit test fail for function B - that can mislead a developer into thinking function B has an error. I argue that a) it rarely misleads anybody for very long and b) if I had function B tested in total isolation with mocks then that error in function A might have slipped through unnoticed. It's tricky to mock up some combination that you weren't expecting to happen.
Obviously in ideal circumstances you would be doing both unit and integration testing. But, you have to draw a line somewhere otherwise you just write tests all day and get no production code running.
1) They run fast.
2) They help us localize problems.
In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly? Naturally, there is a continuum. Some tests are larger, and they use several classes together. In fact, they may seem to be little integration tests. By themselves, they might seem to run fast, but what happens when you run them all together? When you have a test that exercises a class along with several of its collaborators, it tends to grow. If you haven't taken the time to make a class separately instantiable in a test harness, how easy will it be when you add more code? It never gets easier.
- From "Working Effectively With Legacy Code" by Michael Feathers. (A really good book that actually does a better job of teaching design than many a book because like all good teaching it starts with the examples and abstracts from there, not the other way around.)
You need to test both the components individually (unit tests) and their interaction (integration tests).
We use Commons VFS for filesystem abstraction ( http://commons.apache.org/vfs/ ) and our own port of the API for .Net.
In fact we have our own systems API (more Google AppEngine style thank say the .Net Framework or Java libraries) on which we build our software.
When something blows up it can be nice just to get a simple error rather than a 20 page stack trace.
Exactly. Sometimes the benefit of a certain kind of test does not outweigh the cost of distorting a useful design so it will support the test code. That cost might be directly in terms of performance overhead, or it can be a more subtle overhead through (for example) weakening language-enforced modularity in your architecture so you can insert mock/stub placeholders where you want them for testing purposes, which can hurt everything from maintenance overheads to the effectiveness of code reviews.
This makes it more difficult to retrofit to code and is sort of awkward when you are integrating with different bits of code and libraries that don't use DI.
Thus the usefulness of mocking frameworks to "force" a particular class to be a certain implementation for the purposes of testing.
In the real world, software architecture is the servant of many masters, often including multiple kinds of testing, and sometimes you can't do everything that each of them would like all at the same time. For a start, they can be objectively contradictory: every time you add a dedicated test hook or introduce inversion of control to support unit testing, for example, you have also created additional cases that need to be considered during a code review.
However, it's important to remember that all these testing methods and design techniques are merely means to an end. They are valuable precisely to the extent that they help us build software to do useful things, and no more. If they sometimes conflict, that's OK, as long as we favour the one that most helps us to get the job done. However, things tend to go wrong if we start nerfing one aspect just because dogma says we must not compromise on another.
It's also difficult to know ahead of time how much abstraction is the right amount ahead of time.
For example, you abstract disk access through an RDBMS but you might want to change DBMS so you abstract that through an ORM. But maybe you want to switch away from RDBMS altogether and go with a NoSQL solution or do everything with message queues or REST APIs? Should you abstract up another level also?
Well the point of the abstraction is that you don't care about the contents, hence the term "black box abstraction".
It doesn't matter to the interface consumer what the implementation is, just that it does what the interface says it will do.
Our VFS layer has various backends which can be picked based on the client's requirements and this covers REST APIs, RDBMS and filesystems. There are various depths of stacks and abstractions hiding behind all that but to the consumer, it doesn't matter.
If it blows up anywhere in the stack, we can throw 80 levels of stack right down to the container thread if need be, but we only care about the abstraction at the point of failure.
When you integrate some third party black box with a different interface you're probably going to have to write at least one wrapper class.
It only makes sense to unit test code which is in the domain of architecture. This ensures that your domain model is solid.
It only makes sense to functional test code which is a concrete implementation running on top of said architecture. This ensures that the end user sees what they need to see.
One of the often quoted benefits of unit testing is that they run fast. That fact is only incidental. Not to mention the insanity of unit testing as much as you can just because they're faster to run - they are also an order of magnitude slower to write than functional tests.
tl;dr - use the right tool for the right job. Unit testing is for design/architecture. Functional testing is for concrete functionality.
1. What do you mean by unit in unit tests?
An implementation of very basic feature, probably not more than 100 lines of code.
2. What do you test when you write a unit test?
We test the basic correctness, eg. given a url, if it exists assert 200, if not assert 404.
3. What should not be tested?
The inbuilt libraries should not be tested. Eg. If you are using a library say django, you should assume it comes pre tested and should not spend time writing tests for them.
4. What should be the depth of the tests?
There are no rules, but you should stop when you feel like you are humming inception theme.
If you want to read in slight more detail its http://www.blog.fruiapps.com/2012/09/An-intro-tutorial-to-te...
There's a very good sentence in SO thread - j_random_hacker's: "Every programmer has a probability distribution of bug locations; the smart approach is to focus your energy on testing regions where you estimate the bug probability to be high."
So Kent Beck is not skipping some tests because he thinks he's awesome (as jiggy2011 said it sounds like); he does that because he can anticipate, from experience, that a particular type of bug is of very low probability for him, because he knows he's unlikely to make it. Like, for example, I tend not to put + instead of ++ in C++, so I can be pretty confident my code is free of these types of errors. Everyone needs to estimate his probability distribution for oneself (factoring for unknown unknowns) and test accordingly. The more experience one has, the better the estimate.
http://reidburke.com/2012/09/27/write-code-that-works/
Your code isn't uniform, and so you shouldn't be expecting 100% code coverage on everything either.
Instead, assign to particular sections of your code levels of stability, and for the code that has the highest stability/least amount of expected future change, prioritize writing the most unit tests for those.
Hmm, that sounds like the old "You don't need to test , just be an awesome programmer duh!" argument.
The kind of mistakes you never anticipate making are probably the sort of stuff you should be testing for.
The amount of times I've had something fail on an edge case that turned out to be a typo in a var name on some edge case somewhere..
The code in question was only subject to very high level smoke tests that looked like they were basically cherry picked best case scenarios intended to pass.
A unit test would have picked this up before it even got to production.
If typos are even a significant problem in your code then the solution is to type more slowly, have your IDE flag unknown symbols, use autocomplete, etc. As a solution to this problem, strong typing is both disproportionate and inadequate.
At spare time, tho, I tend to be too lazy to write a lot of tests(most of my "projects" are too small and never get released to the public anyway)
How does this compare with TDD?
Has anyone tried both? How did it work out for you? I implemented TDD for a project and I thought it was overkill and took a considerable amount of my time.