Comment by unscaled | Hacker News Reader

Comment by unscaled | Hacker News Reader

Parent Full thread

> Testing third-party code is often well within the purview of first-party integration testing.

Yes, integration tests do tend to cover integration with third party libraries and even entire products (such as databases). But even in integration tests, the third party code is incidental.

For instance, when created integration tests for databases, it's very common to use an embedded or in-memory DB such as SQLite or H2. You want to test integration with third-party modules when it is possible and "cheap", but the highest priorities are testing first-party module integration and having tests than can run fast without requiring a full-fledged replica of our production environment.

If I come back to the GP statement "this only works for code you control", they either meant that you can't use this this techniques unit tests of third-party code or that you can't use this technique in integration tests of third-party code. Either way, it doesn't make sense. You cannot and should not unit test code you don't own, and you're not supposed use mocks (like a mocked clock) in integration tests.

>Yes, integration tests do tend to cover integration with third party libraries and even entire products (such as databases). But even in integration tests, the third party code is incidental.

It definitely shouldnt be. At least, not if you want your tests to tell you when upgrading a 3rd party dependency will break something.

>For instance, when created integration tests for databases, it's very common to use an embedded or in-memory DB

I worked on a project that did this once and they very quickly got blocked writing a test by the in memory database not supporting a feature postgres had.

Result? "Oh well, I guess we dont write an automated test for that." Manual QA's problem now. That's slow.

Realism matters. Sacrificing realism for speed often means you will get neither.

>you're not supposed use mocks (like a mocked clock)

I do do this and it usually works well. Why am I wrong?

>>I worked on a project that did this once and they very quickly got blocked writing a test by the in memory database not supporting a feature postgres had.

>Realism matters

It is often just a small subset of tests which have to be very expensive to build, maintain and run because of an external dependency that cannot be mocked. Mocks work for 95% of use cases and should be used even if it is not 100%.

I never found that to be true and moreover, these scenarios with nonstandard features tended to be the scenarios I was most worried about breaking.

There is also the problem of the in memory database behaving differently when given the same SQL so your test might make it look like everything works while a bug crops up in production.

Realism matters.

> There is also the problem of the in memory database behaving differently when given the same SQL so your test might make it look like everything works while a bug crops up in production.

You've horribly messed up the architecture of your application if that is a problem. (Or you've misunderstood previous comments)

Postgres and sqlite/in memory dbs just behave differently to each other sometimes. Knowing this fact doesn't mean you've messed up your architecture it means that you have some understanding of how these databases work.

Realism matters.

I'd say that in-memory/not-in-memory isn't the big difference - it's whether your database is in-process or not. Even with just a database running on the same node, but in a different process, connected to via unix socket, the context switches alone lead to very different performance characteristics. Actually going over the network obviously changes more. It's very easy to miss antipatterns like N+1 queries when you test on sqlite but run on a shared database in prod.

This is irrelevant for unit tests. Performance testing does not make any sense in build environment, you need to do it in an environment close to production and that’s completely different test automation scope.

You can see this stuff often even in test workloads. But even if you disregard that kind of issue, you still have stuff like needing to integrate networked database connections into e.g. event loops, which you don't really need to do for things like sqlite.

Reply on news.ycombinator.com