Comment by RangerScience | Hacker News Reader

Comment by RangerScience | Hacker News Reader

Parent Full thread

> Either you're writing tests or you're... manual testing

100% strong agree. One of the main reasons I write tests now is just to put the code to use without having to also write all the UI/front-end/usage code.

> unrealistic 100% coverage

I'm not entirely sure how I've managed it (I've got a few ideas), but for the last year or so, I've been able to almost trivially reach 100% test coverage, and without a crazy ratio - one recent project that I actually counted, it was 3 LOC of test code for every line of "product" code. Time spent was more like 1:1, although I wasn't counting, and it's very blurry anyway, and testing manually wouldn't have taken all that much less time anyway.

Note that that's simple code execution - that's not including anything like "100% coverage of possible inputs" - and the LOC is counting pretty formatting lines, etc. I'm at about 4x Rubocop's default maximums on function lengths, and I think abut 1.5x it's maximums on "comlexity" metrics, both for the test and the product code.

Write testable code, and make sure your test code itself is code good, and everything gets better.

(At least, in Ruby)

Edit:

Part of how I got "here", with what (AFAIK) is trivial 100% coverage, was at prior garage startup, we read Martin's "Clean Code" as a team, book-club style. Our CTO led it, so we also skipped some chapters, since we were a Ruby shop so not everything was applicable. There are some really good principles in that book, no matter your language.

100% coverage means nothing. Testing every bit is easy if you ask me but coming up with edge cases takes so much time.

Most people agree that coming up with proper edge cases for unit tests is way harder than writing original code. You're not just writing code that works, you need to anticipate in what way it's going to break.

That means that engineers writing unit tests should be better than engineers writing code.

If the data structures that represent your edge cases are simple, maybe QuickCheck can generate your edge cases.

There are equivalent libraries in a lot of languages.

Hypothesis is a useful one to use in Python. Particularly with its state-based testing that lets you describe actions, preconditions, and invariants. It then exercises the program by calling a random sequence of actions and seeing if everything holds as expected. Oh, trough some sequence the "save file" option ends up not saving the file? Good to know, record that seed and track down the problem using the supplied steps which you can now repeat automatically and manually to see where the problem occurred. Maybe add another invariant after doing some research and find it is breaking earlier in the sequence.

It's like having your QA/V&V/Test team doing exploratory testing but automated.

Was part of a team some time ago where we had 100% FE and BE test coverage. Good suit of E2E tests, functional tests and contract tests. In most cases it took longer time to get the test coverage done than to write the code itself.

Overall it was tremendously valuable as it becomes pretty trivial to refactor the code or even move to a new major library version that was core of the architecture. I cannot imaging nowadays without good tests suit as code needs to evolve and you want to sleep at night as well.

Still, you'll never cover the complex business logic and interactions between many many different services in a large complex system that can be only tested doing manually by a person who knows/looks the domain in a higher level.

Bugs still appeared in the RC, live environment when all the systems started acting together.

From my experience 80/20 rule is pretty good one to have. Getting it to 100% meant to run the code coverage analyser and looking at all the code paths that are not yet handled that in many cases were code paths that never caused us problems in live and were not even crucial when doing major refactoring as it was already covered by the 80%.

Looking back at least for that specific project, I'd say having little bit less coverage would have been much better and would have enabled us to move tad bit faster and test new features on the field to validate if they benefit the business or not.

Nowadays I am leaning towards having a pragmatic view on writing tests.

We've definitely got some test suites that slow us down, but...

...those suites would not pass review elsewhere in the codebase. Near as I can tell, most of the time that it's the case "our tests slow us down" is because the test code isn't held to the same standard (ie, the standards are massively relaxed)

That all said, what I actually want to ask -

In my head, having a good test suite - particularly a BDD-style one, like Cucumber tests - mean that it's easy to add tests to cover things uncovered from manual QA.

Have you found that to be the case? Or, have you found that it could be the case, if the test suites were different in some way?

> 80/20 [and not 100%]

Totes. I'm actually surprised that I've been able to hot 100%; lately it's been like 95% after I bang out the obvious tests, and then there's like one branch that's missing and it's easy to add. If/when it's hard to get that last 10%, totes agree - don't.

I have not done Ruby but I found a lot of my <100% stuff to be weird and unlikely errors - logic error "I violated an invariant somewhere else" type of thing, or syscalls / standard library functions that could error out though one would never expect it. So say I `open()` a file and then immediately `fstat()` it- I don't ever expect the `fstat()` to fail, I'll write the check, log, and failure/crash just in case it does come up, but rigging up a successful open() / failed fstat() for the test seems a bit much.

Reply on news.ycombinator.com