Comment by xivusr | Hacker News Reader

Comment by xivusr | Hacker News Reader

Excellent article and it's perfect timing for me!

In the past while contracting I was usually asked to include in my proposals estimates for tests.

The tests failed to be useful, simply because they were written after the feature was actually implemented! We knew better, of course, but this was done in order to quickly get builds a client could look at.

Then when clients wanted to add/change features guess what got cut to make up for the time? Thats right, the tests!

So the tests were always secondary, and the projects tended to suffer as a result.

Recurring "fixed bugs" cost more than just working hours to fix. In the eyes of a client or customer, they are so much worse than a shiny new bug. Tests can help catch recurring bugs before a client/customer does - and save you not only time, but from losing your customers confidence.

Now, I'm building my own app and I'm using a diciplined TDD approach. I didn't start my project this way as It seemed overkill when it was just me. But I saw early on that to not practice TDD even solo was madness. It is taking longer, but my actual progress is consistent, and I'm already far more confident about the stability of the app.

Tests written after the feature is implemented can be very useful to keep an eye on regression in the case of teams where multiple developers are checking in changes throughout the day and there is a continuous integration build and test system present.

At two companies and across three teams where I've worked as a developer responsible for automation I have found that for integration and system type tests writing tests after feature implementation and properly reporting results following high frequency checkins has been allowed me to have a significant impact on the way we work to the point where some of the leads and management are actually taking testing more seriously than they used to. It's still hard to get rid of the idea that tests are secondary but they're not so eager to cut them as a matter of saving time as before after seeing how they allow us to identify problems faster, and therefore fix them faster.

In this article the author looks he's talking from the perspective of mostly a single developer or contractor but for bigger teams tests written after feature implementation are still important so I would say based on my experience don't discount them. That said, his approach to writing tests is something I agree with, rather than rigidly sticking to a particular methodology he is pragmatic and it's better than doing nothing.

Also he doesn't mention this but solid reporting is so important. I am involved with Tesults (https://www.tesults.com) which is a test results reporting application designed for teams of 10 or more to integrate into their build/test system. If you work for a team of this size and are serious about automated testing check it out. Send me an email if you're thinking of introducing it to your team and have any questions or feedback.

I've mentioned this in a couple other comments here, but it bears repeating here because this comment comes so close to what I feel is truly the most important principle, but falls just short.

Yes, you can construct a regression test suite out of tests written after implementation, and this test suite will hopefully catch incorrect or incomplete alterations made to the software later.

However, the _reason_ it can do that is because these tests will have been written, ideally consciously _because_ of this underlying reason, with the original design of the software in mind; they will be essentially a written record of what the software was _intended_ to do and was designed to do. If one of these tests fails, it means that a design element has been incorrectly or incompletely altered; a well-written unit test of this sort will have an easy to follow flow which will, upon failure, clearly communicate to the programmer which original design element he has not correctly altered and under which specific conditions the error actually occurs.

Writing good tests after implementation can indeed encode a hugely significant amount of information. (Of course, it should also test intended behavior with respect to a wide range of both good or bad inputs, but that is beside the point I want to drive home here.)

Well enough written unit tests can be used as documentation of an application's design!

[N.B. The above thesis really only holds if the authors of the original code write unit tests, ideally as they develop while any design considerations are fresh in memory. As discussed in another comment, sending other engineers back later to try to figure out what the design was and then unit test their interpretation of they think it must have been will almost definitely not give anywhere near the same quality.]

>> Well enough written unit tests can be used as documentation of an application's design!

That sounds really cool. Is there an open source project that you respect that has achieved this, that you can point me to? I'd like to look at it.

Have a look at the unit tests for the components and services in https://github.com/smaato/ui-framework. We designed the test suites to explore and explain how the interface of each module works. So reading the tests is one way to get a feel for how each module is designed to be used.

While a test suite gives you a collection of examples of use, it doesn't tell you the intended purpose and motivation for the software, why it is designed the way it is, provide any argument for correctness or best-, typical- and worst-case performance, or discuss what trade-offs were made, and why. It is often hard to divine the underlying rules from a set of examples, and they don't go out of their way to point out which are the corner cases. There is a great deal of information in the design of any non-trivial piece of software that is not efficiently expounded by tests.

> The tests failed to be useful, simply because they were written after the feature was actually implemented!

I find after-the-fact asserts, unit tests, and static analysis still useful:

- I catch incorrectly handled edge cases that I haven't hit "in the wild" yet (although others may have!) - I think of new edge cases to handle - I can be more aggressive in future refactoring - I can be more confident in excluding the code from my bug hunts

Many of the issues I catch this way are the kind of issues that threaten to turn into really nasty heisenbugs - such as the occasional missing mutex lock. After all, all the low hanging fruit was probably caught when testing the feature locally ;)

The other thing is I tend to be kind of exploratory in style and if I have written a bunch of tests I'm more hesitant to change something, even if it isn't really working.

Parent is talking about functional test cases, i.e. something that can be tested manually from command line. Unit tests are not a functional or integrational tests, so yes, unit tests are useful even after functional and integrational tests are written.

The concept of a functional test seems to vary quite a bit more from engineering group to group than your usage indicates.

Sometimes, some unit tests border on functional tests and vice versa, the distinction blurring between two adjacent test cases in a unit test suite.

And sometimes integration suites end up testing single functionalities at a time, depending on the design and/or requirements of the application, and could reasonably be called functional tests.

On my team, I hesitate to emphasize "functional tests" as a standalone group equivalent in semantic distinction to unit and integration tests; the proper scope of a "functional" test is defined, to me, by the way the functionalities of an application were designed and decoupled, which depends on the particular project.

(If your organization uses functional requirements and specifications for software design, it's simpler - a functional test verifies a functional requirement. Unfortunately, having functional requirements as a part of the engineering process is significantly rarer than it should be.)

I would probably be willing to say that after the fact asserts useful during development should be considered as candidates to be turned into unit test cases, one to one. Few people use asserts, but I don't see why this methodology couldn't be rather effective, given good practices with asserts.

It shouldn't really matter whether the units tests are written before or after. What matters is how testable the source code is, and making sure the units tests add value.

Of course, writing the tests before the actual source code of the feature enforces the fact that it's testable, but it's mainly a side effect, there's nothing inherently better.

The microdesign flaws that stem in TDD (described in [1]) were real in my personal case, and my code is way better on the long term with the units test written afterwards, because I tend to modify my design a lot while writing code.

[1] http://beust.com/weblog/2014/05/11/the-pitfalls-of-test-driv...

Tests only fail to be useful because they were not written with an explicit purpose in mind; good tests are chosen to explicitly verify behavior that is worth verifying.

If you have a good design in mind before starting development, which is, after all, an ideal situation, it could be argued that it is reasonable to write tests after most of the development, not only to verify robustness to bad input as per the usual, but also as a way to encode a demonstration of the intent of the design.

I agree that this is an excellent article. I also agree that the situation you spoke to happens all too often. I recently worked on the backend API for a mobile app. Because the company wanted it done in a month, they said "no tests!" Now that they've released, they want me to go back and crowbar a bunch of tests into place. I can guarantee the next push they have for a release will be under "no test" conditions again though.

Regarding solo TDD, it is slow going at first, but you quickly make up that time later on, when you add code, break your tests, and can quickly jump in and fix the issue. Saves your reputation too, I think.

Reply on news.ycombinator.com