Hacker News Reader

Comment by piker | Hacker News Reader

The temptation since the TDD movement has been that if you can't test it, it isn't "written". I think "legibility" is a great word to describe that desire.

I have hundreds of unit tests for Tritium, but it wasn't until I started writing a blog using it[1] that I started finding qualitative bugs that weren't picked up (and would be really hard to pick up) with unit tests. They were illegible.

I think this is why indy game dev is resilient to market economics where other software verticals become winner-take-all. Game devs are able to make their products qualitatively better because they dogfood so well. They also, as alluded to in the article, have a smaller "enterprise" surface area which requires so much legibility.

Truth is, you need both qualitative and quantitative measures.

[1] https://tritium.legal/blog/eat

View on HN

Test in general (unit tests paticularly strongly) are prone to the streetlight effect. The more trivial or unimportant something is, the easier it is to test it. This creates the temptation to test the easy part rather than what's important. And this can be made even worse if the organization pushes for "legibility" via the easy "legible" metrics (e.g line coverage) rather than hard "illegible" metrics (i.e. a review of the tests by an experienced engineer).

> The streetlight effect, or the drunkard's search principle, is a type of observational bias that occurs when people only search for something where it is easiest to look.

https://en.wikipedia.org/wiki/Streetlight_effect

Game dev in general has a much tighter feedback loop than most software. If you're leaking memory, you're doing that a hundred times a second. If your code is slow you get visual stuttering. If you want performant code you need to think about things like cache coherence, you can't use GC etc.

I agree, but that's really true for questions like "does it run fast"? And those kind of arguably fit into the "legible" camp. Whereas "is it fun?" is a question less able to be formally tested and articulated. It's "illegible".

The tighter feedback loop that helps to answer that question comes from dogfooding.

I agree with you for almost all of this, just a small nitpick:

GC isn't really incompatible with games, for the overwhelming majority of games, it's basically a non-factor in performance if you spend even the slightest amount of time on performance.

It's kind of similar to native code - don't allocate too much, reuse resources, don't leak them uncontrollably.

It's only a problem when you need guarantees (i.e. the game should never drop a frame) but basically no game fulfills that nowadays and it isn't a player expectation.

GC is ok as long as you aren't writing some factorio-like etc. Modern computers are perfectly fine doing shit ton of useless stuff 120 times a second without blinking an eye.

If you're allocating stuff every frame you'll run into problems quickly. Sure, you can use an object pool or arena allocator, but then you're basically circumventing GC.

malloc often isn't fast enough for games either; arena allocators and ECS came to be for a reason.

The truth that a lot of people don't want to hear is that tests are insufficient to prove your code is bug free

When saying these words aloud (or even on HN) many will reinterpret them as "tests are worthless". It's hard to reorient to "tests are incomplete". I'm not sure why this is such a difficult concept to understand. Unless you're omniscient, how can they be complete?

People need to understand, you cannot measure anything directly. You are always doing so by proxy. If you do not know the limits of your proxy then you're only allowing yourself to be blind to foreseeable problems.

So, tests are hints, not proofs. They prove correctness in narrow areas, but not completely. Worse, you cannot test unknown unknowns. Which, as you suggest, is why dog fooding is such an important part of the development process. You aren't just a developer, you need to be a user and an adversary. You should try to see your programs from as many perspectives as possible. This too won't get you to full coverage, but it'll get you a lot further than just tests and it'll help you find what tests need to be created.

If you're just doing TDD, you're just applying Goodhart's Law

I think this is true. I am a big fan of TDD but it remains important to also do a manual test. Often things just don't work out exactly as one would think. This is especially true with things that relate to user friendliness but also with other things.

Reply on news.ycombinator.com