Comment by dmm | Hacker News Reader

Comment by dmm | Hacker News Reader

Parent Full thread

From my experience the downside of functional testing is that it's slower and most importantly when a functional test fails it not obvious what is broken.

The counterpoint I've heard is, "But we run the tests on every commit! If they fail it must be the latest change."

That's a good point and obviously running the tests on commit is a good practice but I still think unit tests have value because:

1) Even if I know the lines of code that caused a test failure I'm still not sure what invariant I've violated. A unit test failing called "EnsureOutputIsEven" immediately tells you what's wrong in a way a functional test "ValidatePdfEmbeddedVideoGeneration" doesn't.

2) Unit tests help isolate problems from miscompilation, platform bugs, and undefined behavior in a way that a functional tests can't. A unit test tells me, "There's something wrong with this specific function" and I can look at the source for undefined behavior and the generated code for problems.

A functional test just says, "Something is wrong somewhere! good luck!"

> "But we run the tests on every commit! If they fail it must be the latest change."

I really want to share a great failure here I've seen:

This was in a django web app, nearly a decade ago. First team I was on professionally, and took around a week to figure out (no one else cared because it was so rare, so I was the one looking for it): memcached would occasionally return corrupted data, that python couldn't un-pickle into an object, because the class attributes didn't match.

Turned out to be because of how our django settings were organized, a large collection of shared settings that no one ever looked in because they were set up once and then ignored, a set of overrides for live, a set for beta, and each dev got their own. The trick was that, to make it easy to set up, the memcached host/port were in the shared, live, and beta files - but not overridden in dev, so all the devs and the CI suite shared the same cache.

This meant that when one dev was changing the structure of one of these cached objects, before committing it all tests relating to it could fail for all devs and in CI.

Ok so you're saying once in a blue moon when a test fails, having small tests will save you an hour. And that's worth the extra time you spend every day writing and rewriting them?

> the downside of functional testing is that it's slower

This is certainly true in some situations, but not all, and I think it's important to make decisions based on the tech stack and tools available. For example, I've found that React + Enzyme + jsdom is a way to write frontend functional tests that are both fast and reliable, so in a codebase like that, I strongly prefer functional tests because they provide much more confidence. I've also found this to be true for software running against low-complexity data stores that are easy to fake, and for things like compilers that are running fully in-memory anyway.

> when a functional test fails it not obvious what is broken

FWIW, I've found that with a good debugger, it's usually pretty quick to start with a functional test failure and trace the issue to the level of specificity that I'd have if it was a unit test failure. But like the other point, I'm sure it depends a lot on context.

> For example, I've found that React + Enzyme + jsdom is a way to write frontend functional tests that are both fast and reliable

I found that tests using Enzyme/testing-library are actually more than 4x slower than unit tests, and give you far worse debugging & increased flakiness.

You can still assert one small thing even with larger tests. Whats the relationship between single field of input and single field of output through multiple objects?

Thats the invariant you test. It's also relationship the business is actually interested in.

Reply on news.ycombinator.com