Proper unit-testing roughly doubles the amount of time it takes me to write a "unit" of code. This is a high cost so I do not have 100% unit-test coverage.
Over time I have developed an intuition about whether I NEED a unit-test for a particular unit. When I think I need tests I write them.
Typically I'm correct about whether the units need tests but when I make a mistake (as evidenced by bugs in the code I didn't unit-test) I go back and write the tests.
This "middle way" works well for me.
[1] - I don't have time to look-up the specific link right now, so you'll need to trust me on this one! :) But I'm sure one can google it. I remember reading it on an IEEE magazine years ago.
Thanks to the version control tools we use this is easy. Adopt a policy where you commit code for a SINGLE reason at a time. Enumerate a few standard reasons such as BUGFIX, FEATURE, FEATURE_AND_UNIT_TEST, REQUIREMENTS_CHANGE, UNIT_TEST_MAINTENANCE. Estimate the engineering time that was spent on making the change. Log that duration. Periodically analyze your performance and see for yourself how much time unit-testing is costing or saving you.
Without solid unit tests backing you, activities like removing "obsolete" methods or renaming classes/modules turn into nightmare with duck typed languages.
You're just exhibiting a basic dislike of dynamic languages. Type mismatch errors are mostly orthogonal to testing.
"Without solid unit tests backing you, activities like removing "obsolete" methods or renaming classes/modules turn into nightmare with duck typed languages."
This is a nightmare regardless of typing. If you remove a method that is still used somewhere, you've broken something, and you may not spot it until after it ships to customers. It doesn't matter whether you have type checks or not. A compiled language that links in those classes and methods at compile time does resolve this issue, but it's not related to types (and you have to wait on the bloody thing to compile every time you change one character in the file). Testing does resolve this problem--but it's not a problem specific to dynamic languages.
If you get a warm fuzzy feeling from type checking from your compiler, then by all means, enjoy it. But there are tools for most mature dynamic languages (Perl and Python for example) that will do that checking without having to write tests manually. But, having never found type errors to be a problem in my Perl or Python code, I've never used them. Some problem sets are more prone to type mismatch bugs than others, and I guess my areas of interest are among the ones that are rarely effected.
How is it not related to the compile-time name resolution and type checking? In a compiled typed language, you would get a compile or link error if the symbol is not found. I regularly write and refactor C++ by inducing compile or link errors to identify the spots in the code where a specific method is called from.
Please re-read, carefully. I said that compiled and strongly typed are two orthogonal language characteristics, and that each of those characteristics solves one set of problems (while introducing some of its own).
I specifically said exactly what you've said about compiled languages identifying missing methods. But it's not because it is a strongly typed language, it's because it is compiled.
What's so bad about having a little clarity in our use of terms and concepts?
The point jey was trying to make is that only languages that are both statically typed and compiled can catch missing methods. Consider for example a piece of code:
a.doB();
If the language is not statically typed, then the compiler does not know the type of "a" at compile time, and therefore doesn't know what class to check for method "doB". This is why even if you were to compile python you would still end up getting missing method error messages occasionally.
We do write tests for this one and we don't have too many problems with bugs.
Tests take almost as much time to develop as the features do. I just wonder if it is worth the time and if other people with startups write automated tests.
http://wilshipley.com/blog/2005/09/unit-testing-is-teh-suck-urr.html
I think what he's saying is unit tests don't add much if you are doing UI type development. It is hard to write unit tests for UIs (web or GUI), so if most of your code is UI code, don't bother.
However, if you are writing a library that will be used by a lot of coders, unit tests are probably indispensable. Otherwise, you would never be able to make even simple architecture or design changes (refactoring).
At my previous start up we had unit tests run on the entire system automatically after every check-in (with CruiseControl) and whenever someone broke a test suite, everybody got an email with name of the offender and diffs of his code against the trunk.
We also had a "beer rule" about violatros gettings drinks for everyone, to make the process a bit more exciting.
As your code gets larger, and as you start to have others working on it (who could break it), unit tests become more important. If you dont foresee either of these happening, then the time investment for tests may not be worth it..
what's the impact of a typical bug in your app: an error 500 page (annoying) or a BSOD or corrupted data (much worse) or a missile hitting a house? these will merit different levels of testing rigor.
if something is breaking, how do you even know? how much time will have elapsed? how long will it take you to find the cause, and is suitable debugging information available for your investigation? how quickly can you come up with a fix, and how expensive is it to deploy it? this is, in my experience, a much more effective place to optimize if you're a web app, especially if you're a small team.
for the non-core pieces of a web app, i'd rather release twice as fast and have logging and monitoring infrastructure in place to find and fix problems quickly (e.g. exception - logged and SMSed to our cell phones) than spend 2x to write unit tests (and maintaining them later.) a nice property is that you'll tend to find bugs roughly in order of decreasing impact, or at least prevalence. however, this approach won't do much against silent failures or bugs that are both rare and high impact.
there are lots of other effective tools: automated smoke tests (by which i mean end-to-end tests that quickly test large pieces of the app) deliver a lot of bang for buck, and root out unintended consequences of seemingly local changes that even unit tests sometimes miss. i also find generous use of the assert keyword finds failures fast/nearer to the cause.
"I don't use it" or "Facebook does not use it". So? What does that prove? Who taught you gentleman, to use such useless arguments in a debate. It sounds like "I do not believe unit tests are useful because I do not use them". Your logic is reversed.
BTW Einstein did not unit test either.
IMO proper unit testing is a skill and takes some time to acquire. Currently I am thinking about good interview questions for our candidates that help us find out how well people understand the reasons and goals of good UT.
Moreover, there are some areas of development that are notoriously hard to unit-test (with good results): Win32 UI is a good example, and even in those cases it's possible to write little, but highly effective UT-code (for instance if you have a unit-test-friendly design in place, like mediator pattern when you can "drive" your UI actions by sending commands to it).
Writing good tests means you don't have to find and read the 200 relevant lines of code.
The quality of the code is a combination of the quality of the programmers and the quality of the processes they use to produce the code. If you have really good programmers, they can use a shoddy process (write, compile, ship) and still produce reasonable quality code; if you have average programmers with a good process (write test, write code, compile, fix warnings, compile, run test suite, fix errors, compile, run test suite, ship), you can produce good code.
What I've found is that whether a test will be useful or not depends on the nature of the code. If I'm writing logic code or building a domain-specific language, an automated test suite means the difference between getting it right quickly and taking a long time to get it hopefully probably mostly right.
Readability is only one aspect of code quality, and code that does something complex and convoluted will never be simple and straightforward. For that matter, complex, convoluted, and correct is preferable to simple, straightforward, and wrong, and that's the sort of thing that a test suite will tell you.
Readability and correctness are orthogonal concepts.
As far as backups go, a backup is only as good as it's age. If a startup is struggling to decide if they've got the resources for writing tests, they probably don't have the resources for maintaining a real-time database mirror either. You can argue that you should always back up before you roll out a code update, but the real world's not always like that.
All you need to do is configure the Point In Time Recovery feature, write a little backup script (mine is 165 lines of Python), and then call it from cron.
With PITR as long as you have the last backup and the archived WAL files made since the last backup you can recover to the SECOND before your program puked all over the DB.
On the other hand, if I knew of a great tool for automated page tests I would use it. I don't think creation of such tests should be hard.
I wonder if the solution is to be careful to write very simple, well designed code.
At inAsphere.com (2000 teen content dot-com), we had no tests and didn't really miss them. Our software was dead-simple though: a bunch of 3rd-party forum and link directory stuff. We got killed for other reasons (i.e. there wasn't really a market and we kinda sucked content-wise).
At Traxit (2000-2001 venture backed startup I worked at, selling remote access software to get through firewalls) we had no unit tests, and it sank the company. We got into a QA death spiral, where every bug that QA found made the engineers introduce 5 more, and the product quality suffered so much that we cancelled our launch. Cancelling the launch is the absolute worst thing you can do if you have venture funding, because the VCs will swoop in, throw out most of the management team, and try to pick up the pieces. Of course they'll fail, so the company folded about 6 months after I left.
At FictionAlley (2002-2005 nonprofit Harry Potter fanfiction archive), I started with comprehensive unit tests, determined not to repeat the mistakes at Traxit. I then got frustrated with how I was spending as much time writing test code as production code, and abandoned them. It eventually launched, a little buggy, and the users were reasonably happy despite some annoyance at the bugs. However, the lack of up-to-date tests has made it very difficult for new developers to come up to speed and change things, which means that imperfections in the architecture have kinda fossilized.
At my previous project at my current employer (2006 IDE plugin), I released without any unit tests. I then spent 2-3 weeks immediately after release writing a fairly comprehensive test suite for it. Overall it's been pretty successful: product quality is fairly high, and the tests have caught several insidious bugs since. I find they're most useful as a sanity check to show me I'm about to do something stupid. Often I'll do something that looks like a trivial refactoring and then roll back when I find out it's made a whole bunch of tests fail.
At my current project at my employer (2007 financial webapp), we have no unit tests. We are getting along okay without them, but that could be because we haven't really met reality yet. When my employer has released previous products (again, without any unit tests or QA), we've had to deal with some pretty pissed-off customers because of the product quality or lack thereof.
At my startup (2007, flash games website), I've had fairly comprehensive unit tests from the beginning. I haven't been running them lately, and they're a bit out of date, again because it takes as much time to maintain the tests as to write the code. While I was running them, they were useful in the same way as above: they keep me from doing stupid things and making silly mistakes. I plan to go back and make them pass 100% before or soon after launch.
If I had to summarize, I'd recommend that you skip them until after launch, and then any time you don't have an imminent feature request, spend that firming up the test suite. That way you don't delay your release too much but they'll still be there for you when you need them, and you also don't start writing them until the feature set has somewhat stabilized. They're a big investment, but they do pay you back.
There is always a balancing act between prefect text book engineering and just making some money!