Hacker News Reader

Comment by ericb | Hacker News Reader

I'm a Rails dev, I do mostly test driven development, but my current bit of heresy is that there are some tests with low or negative ROI that are not worth writing.

Here are some examples: I almost never test at the view level, and barely at the controller level, and instead skip to the integration level. View, and even controller-level tests are brittle. If I'm going to spend time on brittle tests, I'd rather it be at the highest level. My thinking is, if your app can run a marathon, it can probably also walk. I'd also rather my test be married to output than to implementation details, as that gives more ways the code can change and still not break the test.

I feel like some folks view testing like a religion, or code coverage percent as a "score" in the game of coding.

View on HN

I did the same thing until Rails 3 came out -- skipping controller and view tests completely. Now I do Rack-level tests, at least for the non-HTML endpoints. If they are using standard REST conventions, they are even easier to test. I send Rack HTTP request, I get back Rack HTTP response. I test the JSON/XML coming out.

Considering that "web apps" in the future will really mean "mobile apps", I think in the future Rails will mostly drive endpoints and barely serve dynamically-generated HTML, most of the views being constructed out of Javascript/HTML on the browser end. Testing to see that data gets applied to the right DOM element will be easier on the browser side (or even with a fake Node.js browser) if the mocking is done there. I suspect that isolating the views properly from the rest of the app will make the whole stack easier to test and maintain. (It is also easier to find a decent Javascript developer than it is to find a decent Rails developer).

Interesting perspective, but I'm surprised that you're lumping in "controller level" tests (i.e. unit tests) with tests against the view.

While I'm definitely on board that tests against the UI are brittle, I feel like unit testing individual methods and classes ensure your code is modular and functions as designed.

Integration tests are great, especially since they more reflect how a user interacts with the code, but I think you can get a lot of value from unit tests without getting too religous about it.

Aren't controller level tests called functional tests? I'm still new at this, but I thought unit tests are to the model as functional tests are to the controller. And integration tests mimic user behavior.

They are, but the naming convention is a rails-specific quirk.

Usually functional testing is defined as checking functionality vs. requirements, and it is done with something like selenium, or Quick Test Pro when it is automated. And usually, since requirements don't deal with controller functionality but are most often written about the system as a whole, functional testing would be what Rails calls "integration" tests, or Rails/RSpec calls "request" tests.

When the controller tests were named "functional tests" I think integration tests in Rails were not implemented yet, and controller tests were the highest up the stack you could test, so the name made more sense then.

"I almost never test at the view level, and barely at the controller level"

You sound like every Rails dev I know.

It depends on the controller action in question. If it's a boilerplate Rails controller action, then the ROI might not be that high, but if the controller is doing something more specific (assuming it's not doing something that ought to be pushed to the model) then unit testing the action is important, sometimes very important.

Code coverage is pretty much a meaningless metric because it can't say whether the tests are actually good and useful. But it can help point experienced devs into areas of risk that may or may not warrant more attention.

Well, part of that goes to how I code. The controller is rarely doing anything important, more than boilerplate. I push that into the model as much as possible.

If you're going with Fat Models and Thin Controllers, then the controller shouldn't be doing anything more than something standardized. You can probably reduce that code down to using inherited_resources (though with Rails 3, it's just as easy to roll your own). I don't know about HTML endpoints, but I test all my JSON and XML endpoints by sending fake Rack responses. They tend to be highly orthogonal and easily metaprogrammed.

code coverage numbers are ridiculously stupid.

I know you aren't attacking me. I realize that for this simple method, the code coverage is, well, very good (perhaps to a point of being bad). This is an contrived example to showcase the implication of using mocks versus stubs though....or more abstractly to show the difference, and the different way to test, behaviors vs implementation logic. Whether these always have to be tested, I agree is worthwhile to talk about.

I don't wanna become the "Programming, Motherfucker" example of what's wrong with software development :)

There is only one good code coverage number and it's "100". If you aim for 100% code coverage then it really helps make your code better as you must test all the parts, not only the easy ones - that forces you to write code that's easier to test, and "code easier to test" equals "better code" (more sane interfaces/API, decoupled classes, etc. etc.).

However, you shouldn't fall into the fallacy that if you have 100% "code coverage" that you code is really fully tested.

I disagree. Maybe flamebait, but I think 100% is usually either a silly goal, or a good start. 100% for a complex project is basically impossible to achieve (and assumes every area of the code is as important as the next), and for a lot of code 100% is nothing. Consider a function that divides to arguments and returns the result. I can pass in div(2,2). There, done. 100% coverage.

[edit] I forgot to mention what value I do think it has. On teams that I'm on I always say I frankly don't care what the number is, as long as it's going up and not down.

Of course 100% code coverage won't make your code better in every possible aspect, nor it guarantee that tests are perfectly written, but at least a) the tough parts of the code won't be skipped during the test b) you will have cleaner, more usable (testable) API.

I still think it's a bit of a wonked metric. The goal out to be to increase your coverage of possible test cases, not LOC. Since you can't cover every possible test case on any non-trivial project, they must have priority. Priority is probably something like how likely the case is to occur, balanced against the consequences of a failure.

Test coverage is a simple indicator, and simple indicators are handy.. but chasing 100% too blindly will cause you to start seeing every line of code as equal, and they're not.

[edit] - oh, I meant to mention that both of your points (a & b) make sense. I can see how chasing coverage could help make sure you test parts of your code you've been avoiding because they're hard to test. I just wonder if it's the best way to accomplish that goal.

I've seen "100% coverage" specs that managed to run every line of code while only testing trivial, obviously correct, aspects and failing to test any of the actually important behavior. This sort of stuff gives TDD a bad name, but is, in my experience, sadly common.

Also, at least in the Ruby community, people seem to only measure line coverage. Without branch and conditional coverage reports, it's possible to run 100% of the lines but still miss a lot of code paths.

I'll tell you what we do in hardware to get test coverage that is worth way more than 100%: we do fault injection. I have never heard of anyone trying to do the same in software, but it would be interesting.

Here's how it works:

for each statement in your code, break it, then run your entire test suite. If none of your tests have uncovered a problem, mark the code as uncovered. Repeat for each code statement.

It's crazily expensive to compute, but it's so much more accurate than to say that a line of code was hit or not. It measures whether the code had a functional impact on the overall behavior.

That's generally referred to as 'mutation testing'. There are a number of systems that enable this in dynamic languages, the one I am most familiar with is Heckle: https://github.com/ryansobol/heckle

Mutation testing is great, it highlights all the areas where your tests aren't written well, and teaches you to write better tests. I'm not sure that using it all the time is the best use of resources in most cases though...

Amen. Bugs tend to crop up at the interfaces of subsystems, so you're usually better off spending your testing effort at a feature level.

I think tests should not go for code coverage, but for "Cyclomatic Complexity" coverage.

What precisely is cyclomatic complexity coverage?

For instance, you can have a test result of 100% line coverage but if you have composed conditionals and you only check the first part of them... you are NOT checking all the paths of your code.

There are a couple of well-established advanced criteria like decision coverage, condition coverage, MC/DC to choose from if statement coverage isn't enough.

I have no idea what the GP meant, but here's my take:

Set a minimal threshold of cyclomatic complexity. Below this threshold, function are assumed "so simple there are obviously no defects". Above this threshold you need tests, roughly prioritized in order of descending complexity.

It's not a bad idea, but that assumption of simplicity is a doozy. Also, figuring out cyclomatic complexity in Ruby (especially Rails apps) is ... hard. Especially if you want the computer to do it for you.

Reply on news.ycombinator.com