There's always a lot of debate around testing and I'm interested to see how much people do and how satisfied they are with it
IF YOU'D LIKE TO ENCOURAGE OTHERS TO ANSWER, PLEASE UPVOTE - TY
What bugs me:
- Testing frameworks and "best practices" change way faster than language frameworks and I simply can't keep up. What rspec version do I use with what version of Rails? Now I have to use Cucumber? I learned some Cucumber ... oh, now Steak is better. [rage comic goes here]
- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ...
- I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite
- More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful
Feel free to try and convince me otherwise, but I don't ever see myself in the "test ALL the things" camp.
I try to start with just one or two tests to actually help do things that are tedious or require multiple steps. It takes some time to automate a good test but once you do it immediately starts saving time because you don't have to run the same sequence a thousand times while developing. You can think of it more like a macro that saves you time.
Once you write the main test it's easy then to run it with all combinations of good and bad input. By doing that you'll often wind up hitting a pretty good percentage of your code.
Then as bugs are discovered due to unexpected input you can just keeping adding more input situations.
Smart automated testing just takes all that extra test work you're already doing and saves it as you go along.
No need to try to invent extra things to test. You just test what you would have tested anyway by hand.
I used to work at a company where automated testing was sending you emails about what your commits broke. It does help at improving code quality.
1. Write some code
2. Test it
3. Debug your code
4. Test it
5. Debug your code
6. Test it
...
39. Debug your code
40. Test it; now it finally seems to work
If all your "test it" steps are being done manually, you're being very inefficient. A good unit test can actually make development go by faster with the added bonus of defending your code from changes down the road that might screw things up.
TDD isn't "THE" way, but test coverage helps. It's not fun (at least not for me), but it's less aggravating than breaking something 6 months down the road in some non-obvious way. I'm human, so I assume I'll screw something up eventually. Having test coverage helps keep me from shooting myself in the foot later.
You should often be able to at least create an automated acceptance test for what you're doing (e.g., "as a user I want to click this button and see XYZ"). This is usually extremely decoupled from the implementation so it should survive refactoring. So then do your exploratory code, get the test passing, and then refactor, introducing lower-level tests.
If that doesn't seem doable you might be taking on a task that doesn't have a good set of requirements. Writing code without any concrete use case in mind is fun and all but that kind of code should usually be limited to a prototyping sandbox.
Prototype all you want without tests, but once you've settled on a design & are ready to make it production-ready you should spend time at least writing unit tests for your work or (ideally) take what you've learned & re-apply to a clean design written in a test-first manner.
You might think this is a waste of time, but putting code into production without tests is going to give you more trouble in the long run.
It doesn't slow me down, and I can be sure that my feature is there and works even when somebody refactors our software.
1. Write a lot of code
2. Test it
3. It works!
4. Test it
5. It works!
6. Test it
7. It works!
...
15. Test it
16. It works!
17. Are you done?
TDD works fucking great. If you know what you're doing.
Alas, that's a big IF. Most of the stuff I do, I'm just figuring shit out.
Mostly, like when designing a new library, I work outside-in. I imagine how I'd want to do something, writing the client pseudocodeish stuff first, and then trying to make the implementation support my idealized programming mental model.
I end up throwing away A LOT of code. Getting something short and sweet takes a lot of experimentation, most of which are duds.
Though my personal approach of outside-in is trivially like TDD, it's not nearly as rigorous. Were I to be as thorough as TDD, I'd be spending all my time writing tests. Which seems pointless, for code I'm like just going to throw away.
Anyway. Much respect for the guy who wrote that first TDD book. It's one of the few methodological strategies that works as advertised.
That is actually the primary goal of tdd, to free you from the more mundane aspects of the code/run/debug loop. The secondary goal is to give you a good base for changing the code later and finding out what broke, again without a ton of manual actions. But as useful as that is (and it is extremely useful), it doesn't hold a candle to the first benefit.
This is what I find the major advantage of a comprehensive test-suite to be, I don't have to worry as much about breaking any part of the system as a whole - if my suite passes, then I know everything I've worked on so far works, not just the bit I think I changed.
But we must keep in mind that maintaining test code has a cost. Automated testing is not a holy grail and it isn't useful 100% of the time. You should carefully decide what code is worth testing via automation.
Unit tests in general don't catch any regressions, they only help you develop.
Functional tests might be useful, but only if you are testing something that is not likely to change much. E.g.: It is not worth to automate testing the UI if you are going to completely redesign it next month.
Manual testing can actually be cheaper than maintaining test code.
I think this is only in the Rails community, where all new things is quick to be pronounced "the new right way to do things", not just in testing.
> I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite
We do the same (tests for interaction with EC2, Github, and a few other providers). It is more expensive, but we find it more worthwhile too. Normally, 3rd party APIs are insufficiently specified, especially for error conditions. So when we have a failure in production, we can easily add tests to make sure we handle that edge case in future.
Just use Test::Unit and move on with your life. Write some tests. That's what counts.
Imagine the current HTML5 Pandora having bugs with one particular song in Chrome only. How do you test for that using Test::Unit?
I forgot to mention we also use Socket.io for real time push updates to user Backbone models. A good number of bugs don't crop up until another user modifies data and those changes get pushed out to another person.
It's pretty painful to think "oh, this really needs a test, but I haven't got a test suite set up and besides, I don't know to write a test of this kind".
Writing tests for edge cases we see in production is the most valuable thing we do. We use Airbrake to find the bugs, and then we add a test for it, if possible (it's not always possible).
That gives us good confidence that other changes aren't fucking things up. It's also a pretty sane strategy for growing a test suite when you inevitably have some portion of your code which has no tests at all.
This is why regression tests are my favorite type of test. The need for the test has been confirmed by real world usage and once you create the regression test to fail, fix the bug, and pass the test, you won't have to ever worry about users seeing that bug again :)
Good for you. Extremists on all sides are usually wrong.
Shoot for "test MOST OF the things" or "test the MOST IMPORTANT things" or even "test just enough things so that you know if change Y totally breaks MOST IMPORTANT feature Z".
If you have thousands of full stack integration tests that takes an hour to run, you're not going to run them as often as you should be, if at all, ad might as well delete them.
(What's worse is when changes come in faster than the CI system can run the tests. Then you don't know which change broke your build because many changes were tested at once.)
Anything longer than 10 seconds is too long, in my opinion. As soon as you can get up to get coffee while your tests are running, you've lost a lot of productivity. (I recently finished a project where the tests took about ten minutes to run. That meant I could only change code 50 times per day. If the tests had taken 10 seconds, I would have been able to be make 300 changes per day. That's a 6x productivity increase right there.
Fast running test suites are absolutely essential.
It's also nice to be reasonably confident that your commit won't break the build, for which you should probably run a good chunk of the tests before committing.
There are some workflows (Gerrit springs to mind) where you can let the CI server work on your code without breaking anything else, but even then there's a cost to the context switch when a test failure means you have to return to a piece of code you thought you'd finished.
1. Protecting me from stupid mistakes like using wrong variable in parameters, etc. (yes, it is embarrassing to have something like this, but I better be embarrassed by test and fix it before anybody seen it than be embarrassed by somebody else hitting it when using my code).
2. Ensuring refactoring and adding new things didn't break anything.
3. After "hard" bug has been found, ensuring it never reoccurs again.
As for dealing with authentication, etc. - that's what unit tests are for, testing stuff that is under these layers directly. And I don't see it matters what you are using for tests - almost any framework would do fine, it's having tests that matters, not how you run them.
I think you can unit-test javascript too, though I never had to deal with it myself so I don't know how.
This tool is very widely used in Haskell, but it's been ported to a whole bunch of other languages and could make your testing more thorough. In Haskell it's also easy to use and more fun than normal tests, but I don't know what it would be like in a different language.
I feel that way often too but I write test more as a specification for how I want the code to work then as a catch all bugs thing.
"- I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite - More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful"
I feel your pain, I code stuff that use WebGL currently and I find it hard to test that stuff.
About bugs in production, after you find a bug write a test that exercises that bug. Then make the test pass. That way, you're unlikely to ever have a regression on that bug.
For browser-side UI tests, selenium is very useful.
The biggest is that it encourages carelessness. I want to grow more careful and work with careful people, not the other way around. Tests don't seem to make people better at doing science--that is, people test the happy-case and don't try and falsify. Testing doesn't seem to make people better are writing code, and may even be hurtful. Secondly, testing instills a fear of code, like code is a monster under the bed that could do anything if you don't constantly have a flashlight under there pinning it down. Sure, I guess your entire project might depend on that one innocent-looking line of code you just changed, but if that's true, you have some serious design problems and testing is going to make it hard to fix those. Because, thirdly, it hinders design, it's very easy to code yourself into a corner in the name of passing a test-suite.
Related to the design issue is a simple fact of laziness. Your code makes a test fail. Is your code wrong? Or is the test wrong? Or are both wrong? If just the code is wrong, the correct action is to fix your code to fit the test. (Which may have serious ramifications anyway.) If just the test is wrong, the correct action is to change the test. (How many people test their tests for correctness? Then test their test-testing programs for correctness? "Test all the things!" is an infinite loop.) If both are wrong, you have to change both. Obviously people will be motivated to assume that only one is wrong rather than both because both means more work.
In my experience, testing frees you from that fear. You have empirical evidence that you haven't broken things.
My company does Continuous Integration as a service. You would be utterly amazed at how often our customers break their code with tiny innocuous changes.
> How many people test their tests for correctness? Then test their test-testing programs for correctness? "Test all the things!" is an infinite loop.
Try to think of testing in terms of the value it brings to your business. Adding the first few tests to a module has immense value. Adding tests for the edge cases has some value, but you're probably at break even unless it's breaking in production [1]. Adding tests to test the tests? I would say that is valueless in nearly all cases [2].
[1] Bonus: use Airbrake to find the edge cases that happen in real life, and only add tests for them
[2] If you're writing software for cars, planes, medical stuff or transferring money, there is probably value here.
Testing at the behavioral level/systems level/UX level is really verifying a lot more than just "is this code right". It provides a way to check correctness on the specifications, correctness on the behavior, complete coverage of expected usage by the end user, and assures that only the code necessary to get the behavior to work is being written (to name a few).
The carelessness I see are developers writing code without fully understanding the needs of the stake holders. The industry would be in a lot better position if managers/product owners/stakeholders/etc. were expected to provide a good set of behaviors to develop against (as an example, Gherkin or similar tools) before they start pushing developers to "deliver something on time". Note this is at the systems/behavior level and not at the Unit level.
Unit level tests provide robustness. Developers can never assure that software has no "bugs".
Behavior level tests assure completeness. Developers can assure they are meeting the requirements (Developers can't assure they are making what the customer wants: but that is not the responsibility of a developer. That is the responsibility of product owner/project manager/etc. I'm not saying that a developer can't ware that hat, but a developer not wearing that hat should not be held responsible for failings to provide for the wants of the customer).
All that being said, I can not emphasis enough how important I think Behavior Level testing is.
My 3 cents.
Better to write tests to assert something works as expected. Then focus on what you actually want to do, finally returning to your tests and focusing on your changes impact.
If people are writing shitty tests, that is a different problem.
As to your second point, I am fearful of code that does not have tests. I do not know what it does, I have next to no confidence that it does what it is supposed to and no way to validate that I haven't broken it if I change it.
I find the whole pushback for tests automation very odd. Here we are working towards automating some business process, while manually testing that it works. Why wouldn't we automate the testing too? If you are not good enough to automate most of your testing, what business do you have automating something else?
A good reason to write tests beyond QA is to verify your code is at least somewhat modular - being able to get code under test ensures at least one additional use beyond being hooked into your application. For that reason, I would recommend having at least one test for every module in your code. It also makes it easy to write a test to reproduce a bug without having to refactor your code to be testable after the fact.
Think it makes more sense the later stage your startup is where you're more certain of what exactly it is you're building.
It's cool to try to use the API you're building before you build it.
It won't be great, but it will provide some form of sanity checking when you work on other stuff. Of course, it informs the design, which is a very overlooked feature of testing.
Lastly, it provides a foothold for more tests. When you're working on something hairy, there won't be any obstacle to "well, maybe I'll just add this one more test to save me some time".
If you are trying to really quickly get code in front of users, and are working through a lot of ideas that don't end up going anywhere (code that is eventually thrown away), then heavily tested code is probably not the best use of your time.
Once you get a product with some traction, and are going to be working with a codebase for some time (especially a code base that will be growing), heavily tested code is invaluable.
Example: Upgrading a large Rails app (~250k lines) from rails 2.3 to 3.0 in eight weeks. Having roughly a 1:1 code:test ratio allows us to be extremely nimble. It also allows developers to work in almost any area of the codebase with confidence.
For apps that will be around for awhile and will be growing, a large test suite is indispensable.
I'm using rails these days, and I have 100% test coverage on models and controllers (though that really just means that all the model and controller code is executed when I run my tests, these tools can't really tell if you've tested the code intelligently, though I hope I have).
I don't have a full suite of integration tests that validate all of the view logic, though there are some checks. I also have integration tests that validate external dependencies (file storage, database connectivity, etc), though again, there may be some holes.
I picked "all", since that's closest to where I am. But my best choice would be "we maintain a high (95%+) level of testing coverage". I don't think I'm splitting hairs here, because there may be a practical tradeoff between high levels and complete levels of test coverage.
NOTE: "high" levels of testing can mean different things to different people... doesn't have to be 95%, which I would consider to be higher than absolutely necessary. It depends so much on what you're actually testing (anyone who has used a coverage tool knows you can often "trick" the tool into awarding the 100% bar without doing much other than just making sure the tests run the code... which is useful in its way but can let all kinds of errors slip through).
My real goal is to have tests that will sound the alarm if I've done something that breaks the application. I think this is similar to the "smoke test" you're talking about. Don't want to have to fire up the server and walk through all the use cases - it's very useful to have integration tests that will do this instead.
Each test has the opportunity cost of writing some part of a new feature for your customers. But so does every minute spent of fixing bugs that would have been caught with more testing, at a fraction of the cost.
Tests are only marginally important at the time you're writing the code they test. The real value comes months later when something else causes the test to fail, and now you a: know the code is broken, and b: have a clear specification what what that code was supposed to do.
Firstly, even if automated testing isn't appropriate for a particular part of the code, there should still be other forms of quality checking going on that would pick up a broken feature before the code is accepted, and certainly before the product ships. If this doesn't happen, you're relying on a limited set of automated tests as a substitute for things like proper code reviews and pre-release QA, in which case IMNSHO you're already doomed to ship junk on bad days.
Secondly, if you can break one piece of code by changing a completely unrelated bit of functionality elsewhere, you have other fundamental problems: your code isn't clearly organised with an effective modular design, and your developers demonstrably don't understand how the code works or the implications of the changes they are going to make before they dive in and start editing (or even afterwards). Again, you're already doomed: no amount of unit testing is going to save you from bugs creeping in under such circumstances.
Finally, unit tests are not a clear specification of anything, ever, other than the behaviour of a specific test.
Basically, if you consider automated unit testing a substitute for any of
(a) maintaining a clean design
(b) doing an impact analysis before making changes to existing code
(c) writing and updating proper documentation, including clear specifications, or
(d) proper peer review and QA processes
then I think you're suffering from precisely the false sense of security I mentioned earlier. In many contexts, unit tests can be great for sounding alarm bells early and giving some basic confidence, but even in the most ideal circumstances they can never replace those other parts of the development process.
what are you even saying
i dont even
(
More concretely, if you use testing to drive your refactors and architecture--as opposed to, say, finding pain points in normal code or actual design time in preproduction--I would be concerned that you are "guardrail programming", as a gentleman put in a talk I saw recently.
When we drive, we don't have guardrails to bounce us back on the road every time we veer off--they're there to protect us against accidents or when something goes seriously wrong with our vehicle. If you told somebody that you drove from city A to city B by hugging the guardrail, they'd say you were nuts.
Similarly, depending on unit tests to do design is strange--they're there to be sure that your code functions according to contract.
)
Writing tests makes you think about how pieces of your code interact with each other, dependencies etc.
As an example, if you're trying to test Function A and are finding you need tens of lines of setup code to be able to do so, then that would be a warning sign that you may want to think about refactoring out some of those dependencies
Unit tests are THE most overrated buzzword of the last 10 years.
There are many ways we try to improve code quality and make sure we get it right. Automated test suites are only one of them. Software design needs to take multiple factors into account, and letting one of them arbitrarily dominate all others is a dangerous path to take.
I certainly support unit testing, as its essential--and anyone telling you otherwise is bonkers--to ensuring that code follows contract.
That said, if unit testing was great for design but didn't spot errors, it'd be useless. Whereas, if it was useless for design and good for errors, that's okay, because I can do the design work myself.
This realization only made all the other arguments for testing that much stronger.
Many people who claim that test automation is "too much overhead" either don't understand what test automation is or don't understand what overhead is. If you have to test everything in order to change anything you either have a huge manual testing overhead or have a huge quality liability.
Article about the group that writes the space shuttle software, sort of relevant?: http://www.fastcompany.com/magazine/06/writestuff.html
A full run of our test suite literally takes months on a cluster of hundreds of CPUs (obviously, there are also faster versions of the tests which are run frequently). While I have a long list of additional test coverage that I would like to add, what we test is much closer to "all functionality" than it is to "a few critical things".
I'd say that a lot of good responses would have been in between those two.
I think it's safe to assume that anyone who selected "All functionality" actually means "Most functionality". Also I think we can assume that a good proportion of people who selected "A few critical" would belong in the "Most" bucket.
When I saw my codebase has 'all' functionality tested, I mean we don't commit code without tests included too. I think that's a pretty reasonable definition.
double sin(double x) { return x; }
However, even if the sin function is already tested elsewhere, you will still need further testing to ensure that you are calling it correctly (e.g. not confusing degrees and radians).
EDIT: Yes, I read it wrong - clearly need coffee...
That said... that kind of coverage isn't quite as useless as it might seem. If your tests do execute every line, even in a completely contrived way, you will catch a lot if you change your code. You just tend to catch more of the "wrong number of arguments passed to a method" kind of error than "you are allowing the autopilot to try to land the plane 100 feet below the runway" kind of error ;)
However, if you consider functions on the code level (e.g. Java methods) then organisations with 100% coverage will be thin on the ground. If you go further and consider line coverage, almost nobody will have 100% coverage.
A common problem with organisations claiming to test all functions is that they will only test the happy path - there will be few tests for things like unexpected or illegal input etc.
No matter how many times I explain or quote higher or tell them the feature creep is becoming unreasonable (oh by the way, we have 18 products with complicated interactions, not the 3 we asked for on the quote, but we expect to still pay the same), such that I can't possibly write it all and test it all, they just don't listen and they leave me holding the bag. So, while I'd like to do testing, just getting the thing kind-of working isn't in the budget, never mind getting it working well.
Sorry for the rant and... come to think of it, it may be time for a new job.
But... it does sound like you need a new job.
But, are you really going to deliver code that you developed without testing it in some way? You might not be writing test code, but I'd bet you are still doing testing of some other kind.
That said, I subscribe to the philosophy that testing is only there to support the business, not and end in itself. We often prototype features with no testing at all, because they get rewritten 3 times anyway. Often, writing the tests is what highlights flaws in our logic, so without it we would often we flying blind.
Testing slows down coding by about 135% (yes, more than twice as slow), but makes that time back in spades when you have to work on the same code again, or when changing lower layers (models, libraries, etc).
When I write a software package/library, I'll usually test the hell out of it for the very same reason so many others have given: if you're testing in a REPL anyway, why not just turn those snippets into unit tests? Hardly any effort.
But I usually don't bother with too much automated testing for websites or web apps, because (1) it's more difficult to actually catch the errors you care about, have good test coverage and keep tests up to date than it is for back-end stuff and (2) I actually like clicking through my app for a while after I've implemented a new feature or changed an existing one.
Manually testing a web app allows you to catch many different kinds of mistakes at the same time. Almost like an artist looking at an unfinished painting. Does the UI look off? Does X get annoying after doing it ten times in a row? Does everything flow nicely? What is this page missing? Did that customer's feature request you got three days ago actually make sense? Questions you should be asking anyway, even with automated tests. And basic functionality is tested because the underlying packages are tested.
... but then again, if I was writing a website backed by a RESTful API, testing that API is as easy as doing a couple of HTTP requests and checking the responses, so you'd be stupid not to go for that quick win.
So my answer is "We have a test suite that tests all functionality" and "Tests? We don't need no stinking tests." at the same time.
I am seriously considering putting together a "Software Engineering for Small Teams" course or set of articles. With a little bit of expertise, you can inject testing in to most projects, use the minimum of Agile that'll help, and generally massively raise your game - and by that I mean code faster, better, and more reliably, with considerably less stress.
(edited: turns out I forgot which year we're in :-P)
I used to always write proper full-fledged tests. Then I started my startup, building a product in the few hours left after a demanding high-stress job and a tumultuous private life.
Within a few weeks, I stopped writing tests. Within a few more weeks, I turned off the test suite.
I wrote the product, got it working, received market feedback, realized my model was all wrong, rewrote the entire domain model and UI multiple times all to finally realize that my component boundaries were all wrong and intuitively understanding where they should've been.
Now I feel confident about an architecture that will stay stable for 12+ months and each new component I write is properly tested.
In the meanwhile my lack of tests is starting to bite me very slowly, but I find that I'm just slowly replacing all 'bad parts' with properly tested components with clearly defined boundaries, rather than changing existing code.
And in the end I'm really happy that I decided not to test as much. It has it's place but when your time is really precious and you're trying to mold your software to fit the market needs, it just isn't worth it.
I don't know how many others are in a similar situation but, for me, sometimes it just ain't f*ing worth it.
One of the keys to making that work for us is a short feedback loop. We automatically release on every commit, which means every couple of hours. Speculative features get minimally implemented; if they look good then we beef them up more. Our goal is to avoid not just the unneeded tests, but the unneeded feature code too.
I'm personally pretty happy with the testing in that we don't have to spend much time on debugging or manual testing. It's very nice to make a major change, poke at it a little bit, and then ship it with a fair bit of confidence that it will work.
http://www.writemoretests.com/2011/09/test-driven-developmen...
I'm always amazed by how well the whole 'technical debt' analogy holds up. Yes, leveraged development at the beginning is fast, and sometimes a good idea for getting to MVP. But the cost is still there, and will become apparent, and needs dealing with.
Experimental features may be very short-lived, or require extensive tweaks, and the technical debt that accumulates from not testing may never arise over their lifetime. Once you're sure it's going to stick around forever, do it right and cover it with tests.
When you're in this position, your going to get more value out of creating a smaller set of functional integration tests that cover the critical functions of the project. Sure, adding new tests as you add functionality is a good idea, but it's not going to result in total coverage for a very long time.
Of course, tests don't actually tell you these things. But they can tell you that your assumptions were wrong, or that your trivial change broke feature xxx, and that's crucial information to have.
I appreciate the idea here, and I've done the same in certain circumstances, but typically that means writing tests for bits of functionality that I need to touch.
Without good automated, easy-to-run tests, you're going to blow more time fixing bugs and ad hoc testing in the long run.
Thinking you save time by not testing is a lie on all but the most trivial of projects.
Does an ER surgeon always have the time/bandwidth to scrub hands before surgery?
I feel that it is unreasonable to expect that you will be able to pick up any code base and immediately write sufficient tests to get coverage on a majority of the code base. Speaking from my experience picking up old code bases, just being able to write isolated unit tests would require refactoring most of the code base, which is typically not something you will have time to do before you're expected to do other work.
I can't think of a single manager that I've worked for who would accept me saying, "it's going to take me 3-6 months of refactoring & building tests before I can start fixing bugs and providing enhancements."
I can't think of a single developer I've worked with who would try that approach.
When a bug is identified in a project with few-or-no tests, the approach that I usually see taken is to write some sort of large, slow integration test that exercises bug, then fix that. That allows you to prove that the bug exists and prove that the fix fixes it, at least for the documented case(s).
There's no reason to cover an entire legacy code base with tests if you're only changing a small portion of it.
My area of expertise is adding tests to legacy codebases. Obviously you're not going to hit the whole thing overnight. But that's no excuse for not having /any/
You may say "I'll thank myself later", but in this sort of business there won't be a later if we're not fast enough. It's a lesser of evils thing.
It would be nice if testing were a faster thing to do. The faster you can do it, the lower the threshold would be for this sort of a judgement call.
*2012
With a small codebase that you know every inch of, its easy to test most of your interactions before you push something live, but when you get just one order of magnitude higher you start seeing how easy it is to write code in one section of your app, test it rigorously, but not catch some subtle breakage in another (seemingly unrelated) section of your app.
In production software, especially if you have paying clients, this is simply unacceptable; which is why I've recently been boning up on BDD, TDD, and continuous integration and am trying very hard to slowly integrate them into my development process.
To one of the comments before, in my experience, automated testing should actually makes you bolder with code not more fearful. We have this codebase where I work that is a frickin mammoth of interrelated modules and its so scary to go in there and add or change something, because I just know something else is going to break and I'm going to be stuck fixing it for days after I made the first edit.
This is the other reason I started exploring automated tests ... because I realized that if I had a test suite that could catch regressions when I refactor code, then I could actually spend more time whipping old code into shape instead of patching it up until such a time when I'd be able to just rewrite the whole thing.
It took me a lot of effort to learn it properly, I have read many books about testing, have read the tests of plenty of open source software to see how others do it and I wrote thousands of wrong tests until I got at a stage where I can say I have mastered testing.
I was always fascinated about test driven development, but to be honest, it does not work for me and I seldom do it. In most cases I normally write new functionality, then I describe the behavior of it and finally do some refactoring until the code quality meet my needs. When you can refactor a class without breaking a single test, you know you've done it right.
It's important that you find your way and don't try to follow the rules from others. Take your time, mastering software testing is a complex discipline and it won't happen overnight.
Even with a high level of test coverage, I always encounter new bugs when using the software. But after fixing it and adding some tests, I know at least that I will not see the exact same bug again.
I believe that writing tests speeds up my development. This may seems illogical at first, but without the tests my development would slow down with increasing complexity (Lehman's Law), and instead of adding new functionality I'd find myself fixing old stuff. So testing allows me to manage a large and complex codebase, it allows me to do a complicated architectural refactoring and I know everything important still works as expected.
[1] Initial stage where we are trying to make things work. At this stage code base is very small < 1000 lines. This is like prototyping. It works with limited functionality. No tests needed at this time.
[2] Heavy development phase. At this stage, we have proved the concept. Now we are adding a lot of new features. We identified some features as must have. Also, code is getting re-factored/re-designed based on what we learn. At this stage, we add tests for the must have functionality. Thus, we can ensure that important features are not broken by newer code.
[3] Mature phase. The code is mature. Most of the features are working fine. Code base may be large 100000+ lines. At this stage re-factoring/re-designing is not easy. Mostly incremental changes are happening. At this point, we should have upwards of 70% code coverage. Typically, the test code will be more than the code when we have 70%+ code coverage. But, it is very important to have tests, since it ensures that all features are tested even when a minor code change is made.
Perhaps this is because I am in the enterprise development world as opposed to the start-up world.
The cost and frustration involved in delivering a critical bug into a QA or production environment is much higher than the cost and frustration of writing and maintaining tests.
Every action in business has a cost associated with it. The more people involved (customers, UAT, Managers, etc.) the higher the cost. The sooner you can discover the bugs and fix them the less people are impacted the lower the cost.
This is how you make yourself as a developer more valuable and justify your high salary/rate by ingraining habits into your daily routine that reduce costs for the business.
In this I also imply non monetary costs, like the personal costs involved in asking a VP to sign off on an off-cycle production release due to a bug that could have been identified by a test prior to the integration build.
How?
People who don't really write tests will tell me that the advantage of unit testing is being able to see when code changes have broken stuff (which is fair enough and true).
Those who regularly write unit tests will probably bring this up- but often their first point will be 'It helps to structure code properly, make me think about dependencies, modularise code appropriately'
There are people who could write quality software with good test coverage without following TDD style.
I wish someone could make a simple service that allows me to set up my web app, set up test parameters that it tests each and every time, and tell me if it failed or not. I want to automate my babysitting.
There is some reason that your developers aren't engaged in the work. Figure out why they don't care about working code or the user experience and fix that.
If you plug the obvious holes, you won't have fixed your quality problems; you'll just shift them to the places where you won't notice them right away.
I can't agree more with wpietri above. There surely is a non-technical problem at play. That isn't to say that you shouldn't try to automate your babysitting, but if your need for babysitting is that severe, you have other problems.
Their landing page is a little weak but I dig their UI for setting up tests.
you can upload a bunch of Selenium tests, indicate when you want to run them and we'll send alerts if a test fails.
Just as an example, how do you test a parser that processes large amounts of sometimes sloppy semi structured text? Whether a particular defect should be classified as a bug in my parser or as a rare glitch in the source data is undecidable until I know how often the defect occurs.
What I need is a kind of heuristic test framework that makes sure the parser doesn't miss any large chunks that I only find out about weeks later if at all. I cannot supply individual test cases for everything that could possibly be found in the source data.
Perhaps not, but you can supply test cases for known problems you might encounter, as well as ones you've solved after they've been encountered.
I am not automatically testing that, perhaps I am missing something, but automating the steps to log in to Facebook and revoke the token and then also making sure that SendGrid sent the email correctly just seem impractical.
But most of the tests are more fine-grained. So in your example, I'd test the core logic against fake Facebook API responses and a fake outgoing email call. That lets me easily test some of the weirder cases. E.g. if Facebook breaks, will the job skip that user and keep going rather than blowing up?
When you commit to a personal clone of mainline and push it up to the server Jenkins picks it up, builds it, and runs the tests, and if there are any failures notifies you over Jabber and or email to let you know it is broken and for you to go look at it.
We also integrate Jenkins with JIRA, so as soon as Jenkins builds something, pass/fail if there is a JIRA ID in the commit message a comment is automatically added there as well, which if people are watching the bugs they will get notified about.
This effectively allows people to see how they are coming along in terms of their progress and lets them see when stuff is broken almost instantly. Automated builds are nice because we can distribute the builds across a variety of different environments at the same time to see that if maybe something worked on Mac OS X that it doesn't build on Linux, well that needs to be fixed.
It definitely has made me code more defensively, nobody wants to have your Jenkins build show up as red on the status board, and nobody wants the extra scrutiny on code review when asking to merge something back into mainline. So far it has worked fairly well with most developers doing testing.
http://www.bnj.com/cowboy-coding-pink-sombrero/
(article's not mine, but might as well be)
That said, I think TDD is trendy-consultant-crap. Writing a test before your write the code only works for simplistic code that doesn't need much testing and probably won't produce the right test for your code once you have written the code.
Also, for code I've just written, a ad-hoc manual test using the GUI is often much faster than writing a full test and I likely wouldn't ever need to run those tests again. The test suit takes quite a while to complete and if I could add every manual test I've ever run, it would take absolutely forever.
Something like "Zen Test", which runs the relevant tests in the background on code being changed sounds good but I don't think there's anything like it for c++. I'm a bit doubtful it could work on complex code in any language. A lot of R-and-R magic sounds like its creators never went code involving one model method, one controller method and one view method.
So: what about GUIS? How does one formally describe pixels displayed on a screen in a way that captures their ability to correctly communicate with the human intended to view them? So that my notice
Please turn off the foo within bar
Please turn off the foo
How can you refactor safely without tests? You can't. How can you safely upgrade your tools (which often change in subtle ways), without tests? You can't.
"Every programmer knows they should write tests for their code. Few do. The universal response to "Why not?" is "I'm in too much of a hurry." This quickly becomes a vicious cycle- the more pressure you feel, the fewer tests you write. The fewer tests you write, the less productive you are and the less stable your code becomes. The less productive and accurate you are, the more pressure you feel. Programmers burn out from just such cycles. Breaking out requires an outside influence. We found the outside influence we needed in a simple testing framework that lets us do a little testing that makes a big difference."
Quote from: http://junit.sourceforge.net/doc/testinfected/testing.htm
When you work with a team of people that didn't understand what to test you end up with really bad tests that add very little value. Do you delete those test? Write sane ones?
When you end up with a legacy code base where doing something like functional UI testing is easy but doing unit testing on the actual code is almost impossible, do you even attempt to unit test it?
If you see a piece of code with that must be rewritten, but unit testing it costs too much time, do you simply start writing tests what you think the assumptions were and then just go about with the rewrite?
In the end I see a huge value in testing what you write, and being automated is preferred. My problem becomes picking up something else that was clearly done in a misguided fashion and reliably rewriting or refactoring it. I know there are probably some guides/books out there that demonstrate it so any suggestions are welcome.
It is really natural to me, because part of developing idea is a research. It often includes doing some test of how particular library works, what format of data expected, etc. I always was finding myself doing small isolated programs/scripts to test specific question I have about it. And now it was so natural to start using TDD approach.
As other commenters noted, due migration of much logic to JS (client) side, testing it together with server app might be a challenge. For my particular case I "solved" it by using V8 library. I am developing with Perl language and there is great libraries available on cpan. V8 and pure Perl ones. I am using V8 for performance reasons (doing encryption), but before I used pure Perl JS library and it worked perfectly too.
So if your language of choose have libraries to hook into one of JS libraries, I would highly recommend to try to include JS tests into application tests kit.
Stay thirsty my friends.
To all language designers, there is a HUGE space for a better scientific language. Make it easy for Matlab users to understand, but include better encapsulation and library support. Tie in testing and proving from the core.
It should be a one-liner to import a CSV file and do a least squares regression on different columns.
It should also be a one liner to open an image, compute it's 2d FFT, and display it.
It should also be one-liners to do numerical quadrature integration, compute the solutions to some ODEs, and maybe even backpropagation training of a neural network with just one layer.
It's also pretty notoriously difficult to install, especially if you want LAPACK/BLAS. I wasn't able to get it running on many of our servers for that reason and had to revert to Matlab.
I almost said "no way!" then remembered how difficult it was for me to get Scipy 0.9.0 installed and verified via tests.
Still, it's like Matlab ("it" being Num/Sci/Matplotlib) plus all of Python. That's a significant improvement over Matlab if you can get it installed.
Also the practice and triggering the of automated tests regularly (continuous integration) and tracking the time it takes for the tests to run helps to detect early in the development cycle if any of the changes made were suboptimal. All environmental factors being equal a new small feature implemented shouldn't drastically increase the time it takes to run the test suite.
90% of the testing we do is actually on the stored procedures and the general framework. The reasoning here is that these areas have to work right and therefore we have to get this right all the time. Workflow and the like is more fluid, less easily spec'd out, and the like. Test cases aren't as meaningful there but we do have some.
Seriously though, for small web projects I usually aim for 100% unit-test coverage on the models, 70-80% on controllers, and then depending on the application jasmine or selenium to verify the UI components are happy.
For larger projects, add in more integration tests ( models -> controllers, controller -> views ) and on something like mechanize to do full stack tests ( models -> view ).
Additionally for either small or large, running some sort of lint/static analysis at the CI can be beneficial.
It is the CSS, or failing that, the interaction between Javascript and CSS, which I haven't seen any way to test automatically (such test would be able to answer 'given this code, does the resulting DOM look like picture $N').
Usually when there is something wrong with the Javascript it blows up in our faces.
So if anybody knows of a testing framework that can do this, please tell me about it.
* tech stacks evolve changing the amount of testing that is needed => most of the stacks allow to only focus on the "meat" of the logic, rather than things like integration (Spring Integration / Camel), network (Netty), cache (Redis) or even data structures (various language built ins).
* human is getting better with years of coding => I spot flaws and mistakes during code reviews N times faster than I did 10 years ago. I code in little pieces (usually functions), which "talk" back to me immediately even before they are finished.
* REPL is getting really good => Clojure, Scala, Ruby, Groovy, etc.. REPLs save lots of time and prevents mistakes: where a 5 minutes REPLay session reveals a nice and polished approach a lot quicker than a "let's try this / now rerun the test" formula.
* Domain knowledge and "'ve done this exact thing before" greatly impact amount of testing needed => e.g. deeper domain knowledge allows for [better] tests, while no domain knowledge requires lots of prototyping (even if you think it is the "real thing" at first, it is not, it's a prototype), and would greatly suffer from a large number of tests, as most of the time will be spent rewriting test instead of learning the domain.
In the end, the rule of thumb I always use is "do whatever makes sense". I don't buy TDD, ADD and other DDs. They are fun to read about, but they are too removed from the "real thing". If any DD term is needed, what I use is MSDD => "Making Sense Driven Development"
If you have something really long running (eg you make a database and have a two week test), then you may be able to minimize your test (possibly automatically) and use git/hg bisecting to find it.
Fuzz testing (finding holes in your code) can be run separately, and again, you can find the root cause through minimization and bisecting.
TeamCity is good for automating with both kinds of "triggers".
Integration tests. Tick
Automated acceptance tests. No Tick.
Tried to sell concordian as a framework to support BDD - but that is a hard x-discipline change which would have required more effort to push through. So as a short term measure have started to write/express unit tests using a standard BDD style - GIVEN x WHEN y SHOULD z. This has helped to assign value to each unit test. There is now a direct connection between the test name and acceptance criteria specified in a user story.
We have a test framework and a devoted team of people dedicated to encouraging the use of said framework but the rest of our engineering staff don't get it.
So, I test things that matter and don't change too often - core business logic.
100% test coverage is just a goal, a bar to aim for.
And I'm totally with Zed Shaw when it comes to TDD - not worth it when you're still trying to get a full understanding of your problem domain.
We have a lot of catch up to do right now, but i think thats what "startups" often do. We will catch up with the tests in the next month or so, at the end of the day I know perfectly well that without them pivoting and making invasive changes will simply be next to impossible.
Do you think the effort of maintaining all those tests might not have paid off? This question is very tricky to answer.
We write and run so many tests that it is a full time job curating the test suites that should be run prior to code delivery. Basically, if you don;t like writing tests you will be miserable at our shop.
The tricky part is keeping testing standards consistent when you get beyond 30 or 40 developers.
Developers tend to be more opinionated about testing practices than even editor selection and curly brace placement.
The tests would go something like this: 1: SmallCheck exhaustively tests the small cases 2: JUnit/TestNG tests the main use-cases. 3: QuickCheck produces a lot of random tests and hammers the APIs.
Sadly (for Java at least) this appears to be a rather difficult ask.
When you are writing any code, you should try to predict if what you will gain out of it will be worth it. In other words: evaluate the risks of anything you do in your life.
Sometimes it can be tricky (replicating the conditions of their data set comes to mind), but it's quite good for preventing regressions.
That said, they can give you a false sense of security. If your test is wrong, it can allow bugs to slip through the net until your user picks them up at the worst possible time.
I believe this transformation is entirely to do with the fact that I paired with a brilliant developer every day for 6 months who really helped to answer all my questions and show me how to test a variety of different things. I truly believe that unit testing (and testing in general) is a hard thing to grasp without being able to learn from someone over a long(ish) period of time.
I realized that I hated testing because I didn't know how to do it and wasn't good at it. I also didn't understand what the essence of a unit-test was; my tests would often cross multiple integration boundaries (ie: hit the db and the server) and were really more like bloated integration tests. Once I had sorted that out and was able to see a variety of techniques for testing specific scenarios I realized that I actually enjoyed testing and the satisfaction of knowing my code was covered against defects started to be a big motivator.
To answer your specific question about JavaScript testing, I've been using JasmineBDD[1] for the last 2 years and have found it a joy to use. It really makes testing things easy and has tools that allow you to isolate your tests down to the individual units.
Sounds great. What else did you learn?
Here's what I would have clicked, if present: We have a test suite that test a lot of things, but probably only represents %75 coverage at best. We'd like to do more testing, and we're continually adding more, but the biggest barrier is cultural.
The term "test suite" seems to refer to formal testing techniques like creating unit tests and the like. I don't do that. But I do test my program on every functionality by running and checking if it does what it's supposed to do. Does that qualify as testing?
I find my desire to work on those projects directly proportional to test suite coverage. Once you start writing against automated tests, there's no going back...
(Sorry for the obligatory meme reference)
we even use a subset of those tests in production to make sure all sub-systems are working, not just a ping to the api.getsocialize.com domain which isn't sufficient.
When I first started testing I was rather skeptical. But now that I do it, I wouldn't code without it. There are so many other benefits of testing like cleaner code, incremental builds for native apps, and just general confidence in your deployments that allows us to deploy anytime of the day without worry.
http://troll.me/i-dont-always-test-my-code-but-when-i-do-i-d...
I don't know how anyone can move forward in a long term application without having regressions done for you in the form of testing.
We've a code based which is a mixture of javascript MVC (Backbone) and PHP (Zend)
A healthy attitude to unit testing and dev-ops has saved our back more times than I care to mention.
Also - it's a very useful way to "train" new developers. Spending 2-3 weeks writing tests is a great way to get a feel for a) the code b) house style where new developers can be immediately productive - without risking touching production code on day 1
Whenever I deploy new code, I make sure I pass all the unit tests, but then I watch the monitor the dashboard and incoming requests to confirm.
My company finds it worthwhile enough that we are never happy with what we have. We could always write better tests, but we need more engineers than we can find to hire. So we are unhappy with the current situation, but are working to improve.
That's where our dev team is really at.
Just kidding, we have a test suite that tests the things deemed critical and some other stuff.
All important features through functional tests
All critical path features through endurance tests
Ad Hoc + user feedback for the rest
If you can't automate your deployment, you can't rapidly iterate.
I wish I wrote tests more. Maybe I'm to impatient. :(
Take the fortune 500 approach. It's not a bug unless customers complain.
http://www.amazon.com/Facts-Fallacies-Software-Engineering-R...
lulz