Hacker News Reader

Comment by seanmcdirmid | Hacker News Reader

Couldn’t you also test the recovery logic with a hasher that yielded collision more often? It’s like take your system, make it worse (use very dumb hashing), and see how it degrades and/or breaks?

View on HN

My preference is always to actually test parts in as close as possible to the real state, avoiding mocks unless they're necessary. The more "fake" the system is, the less it tells you about how your software actually works.

So since I could generate collisions for the real hash, that's what I did. If it had been infeasible, I would of course have found a different way to test.

This particular software was written mostly in C (today I would not write it in C) which makes it even more likely that introducing never-used-in-production behaviour unexpectedly perturbs the system. This same system for example ran into an obscure (and since fixed) libc allocator bug, and a defect in the Linux kernel, because we were really kicking the shit out of the memory mapped file I/O with what we were doing.

The only high fidelity mantra in testing is a bit misguided. If you only test your system in as close to production state as possible, you are never going to encounter stress states that would only very rarely appear in production. Seeing how your components independently deal with less ideal components in the system can tell you a lot about how they might handle rare but bad cases in production, and allow you to over engineer them a bit more to avoid that.

> which makes it even more likely that introducing never-used-in-production behaviour unexpectedly perturbs the system.

Isn’t that a good thing? Isn’t the goal to deal with the unexpected in the first case?

Tests come with a non-zero cost. They're code that needs to be maintained. Often complex code, with poor documentation. I don't think the value provided by testing states that you will never actually encounter in production outweighs the cost of maintaining the tests.

To achieve “defense in depth,” testing components in degenerate non-production situations (and then over engineering them to gracefully deal with those situations) is the only way to get there. Of course, we aren’t building bridges, so maybe it’s less important.

Thats not really a "defense in depth" situation.

Defense in depth is regarding layering different types of security, not targeted testing.

https://en.wikipedia.org/wiki/Defense_in_depth_(computing)

It's an analogy. Multiple layers of defense against bugs.

This approach works well until you do encounter a situation like that in production :). It's worth testing conservatively because our understanding of our systems is imperfect and because assumptions that may be valid now are likely to change in the future. It's especially true for code that might operate at scale or has security considerations, because both of those can magnify the effects of conditions that occur with arbitrarily low probabilities.

Personally, I see tests as not just a way to prevent bugs, but as a way to learn about the system being tested—both how it behaves now and, continually, how it behaves in future iterations. In this specific case, understanding what happens if two URIs hash to the same value even if they don't "really" do that is an important bit of information: it tells us not only what would happen with a real hash collision but also how our system would behave if there were a bug either in the hashing algorithm itself or in the code that connects the hashing algorithm to the component that uses the hashes.

What happens if we introduce a caching layer between where the hash is generated and where it's used, and there's a cache invalidation bug? I don't know how realistic that is in this specific hypothetical, but I'm sure there are other performance optimizations with the same bug potential that I'm not even thinking of. Changes like that can lead to absolute debugging nightmares! Testing even "impossible" edge cases helps catch this in a systematic way, without needing full knowledge of which conditions might matter.

Of course, none of this talks to the trade-off you pointed out: test code does have a real cost. That is absolutely something to consider. I just don't think the answer is to draw a hard line at "don't test situations that can't come up in production"—and, perhaps naively, I think the "real" solution is less about deciding what is and isn't worth testing and more about reducing the costs of testing more. Write application code in a way that's easy to test and treat your test code as code—keep it clean, include comments, and invest time in setup and tooling to make your tests as easy to write and maintain as possible.

> The only high fidelity mantra in testing is a bit misguided.

That’s understating it. It’s incompetence because it’s an excuse to not test a huge surface area of edge cases.

> My preference is always to actually test parts in as close as possible to the real state, avoiding mocks unless they're necessary. The more "fake" the system is, the less it tells you about how your software actually works.

I've heard this before, but never actually had that problem with fakes in practise. Where can I read more about it?

Have a production outage. See the code in question is 100% tested. Scratch head a bit. Vow to fake less next time.

Supposedly the software problems with the Starliner spacecraft were also caused by extensive unit testing under test harnesses without integration testing, as a visible example that reached the news.

“Test as close as possible to the real state” makes sense as a rule of thumb, but I’m having trouble coming up with a reason in this specific case how mocking the hash function could go wrong. Assuming your testing/mocking library is reliable, this should be testing precisely what you intend to test. I guess it’s possible that your “more real” test relies on some hidden behavior, like how your real hash function utilizes system resources, which could affect production, but I don’t know that tests which are testing things you’re not even aware of is a great thing to advocate for.

The point of the principle is that you don't know how the mocking can go wrong. Assuming your testing/mocking library is reliable gives you one more thing that can unexpectedly go wrong.

Of course, sometimes getting verisimilar test cases is difficult enough that you judge it not worth the effort over mocking it up and dealing with any issues with the mockup, foreseen and unforeseen.

Why not just use a hash function that special-cases two test URLs (or a whole class of test URLs) and produces collisions for them? You can use that same hash function in production.

That’s a stupid principle. If you don’t have your interface well-defined enough that switching out the hashing implementation is difficult, your code is too tightly coupled.

A test handing a hash collision is not a test of the hashing algorithm so you should be able to switch out the algorithm to something super predictable. This isn’t a discussion about mocking frameworks - you should be able to do with by just plugging in a different hash implementation when you construct the test.

These types of blanket rules are useful to ensure a 100% accurate test case. If an error in production costs me $15 Million I don't really care if the engineer wastes a few days to follow a dumb rule.

Also I can employ a $75k - $100k/year engineer and say follow this protocol for testing. If I want someone who can optimize test case 100% of the time with zero mistakes ever then I might have to pay $150k - $200k / year.

It ensures that if someone changes the hashing algorithm, your test for the collision mitigation now doesn't test that anymore even though they should be unrelated.

After going through all that effort to make the test less "fake", I expect tialaramex verified the collision still happened as part of the test.

Yes, actually we also verified that all the hashes were exactly what we expected, because we were using this for content based addressing. "The hash has unexpectedly changed" was a fatal build error.

If we wanted to change how the hash is calculated all extant instances of the system need to dump to backups, then be painstakingly restored after updating. Possible, but certainly not something you'd be doing for giggles.

That’s not useful. A swapped out hashing algorithm would have been proving the same thing and would have allowed you to test important things like 3+ collisions and two sets of collisions.

Anyone who demanded the real hashing algorithm in this case doesn’t understand how hash collisions work and what problems you need to solve when using hashing. Your “100% accurate test case” serves very little value.

It’s the same as requiring a real person perform the checkout workflow in your shopping cart integration tests. It’s “100% accurate” but it’s a massive waste of resources and provides terrible coverage of the many other edge cases.

I think you generally need to either assume or establish that your testing environment tends to be sound, just like you need to assume or establish that your programming language/compiler/computing environment tends to be sound. Your application code and test code itself probably shouldn't be compensating for or attempting to avoid known or unknown flaws in your testing environment.

How do you establish that the testing environment is sound besides having defensive tests? It doesn't necessarily have to be the case that all your tests are like that.

Or, even simpler, just stub out your hash function and to something that always generates the same value and test with that.

Seems like that would make it less of an integration test

They said it was for unit tests.

they mention this was a unit test, for which stubbing the hashing function would have been totally fine

This!

Take over a project with thousands of integration tests.

I trust none of them. Why? Because anything hard to test has been mocked to oblivion.

I have api endpoints that have massive complicated integration tests. I eventually just got source for every project in company and checked to see what they actually did with API.

In this case, I think mocking is the right thing to do. You want to know that you can handle the case where two hashes collide. But for this purpose, you don't care at all how the hashes were generated.

If the website changes its URI hash algorithm, the "honest" test that just supplies two URIs that happened to collide under the earlier algorithm will suddenly become worthless. The mocked test will continue doing exactly what you always wanted it to do -- exercise the "what if the hashes collide?" case in your code.

You still have to reason through “what if it collides with the real hash function?” How do you know it will behave the same if you don’t test it?

(I can sympathize with both approaches and don’t think either one is better.)

I have the inverse - I started at a place that has hundreds integration tests that require a database, k8s, the whole shebang. No mocking anywhere.

Tests take an hour to run, are extremely fragile and flakey. Getting your local dev machine set up to run even a subset of the tests is a half day ordeal, and it can change periodically.

If you have external dependencies that don't version their APIs well it may be inevitable that you need to run tests against them. But for developer productivity you should have a fast/non-flakey path to run local unit tests with no network dependencies. It's also great for working on a plane.

> Because anything hard to test has been mocked to oblivion.

Agree. Not an integration test. If it has mocks in it, it is not testing the INTEGRATION of/between components.

Mocks are for Unit tests. Where you want to isolate testing one single UNIT from all the other crap it integrates with (which have their own unit tests) After all these unit tests run, you test the whole shebang together with integration tests.

I was wondering why GP couldn't have swapped out the hash function for one that just returns a fixed value, making everything a collision. Then it would just be a matter of verifying the system still works under that condition (ignoring the performance hit).

You could even measure the performance hit from that degraded condition and create a heuristic to warn you if something is wrong with the fast path.

Reply on news.ycombinator.com