How to Survive Your Project's First 100k Lines

The sections "If you think you're using enough assertions, you're wrong" and "Bonus: Sanity Checking" can be considered a subset of "Design by Contract"[0].

Regarding:

  However, comments can become out-of-date, and you might
  not be lucky enough to stumble across the right comment
  before you embark on a refactoring adventure.

Regarding "Prefer end-to-end tests":

In my experience, this leads to unit test atrophy, degrading system verification to rely solely on integration tests ("end-to-end tests"). This has often resulted in higher coupling and always resulted in interminable build times.

0 - https://en.wikipedia.org/wiki/Design_by_contract

1 - https://softwareengineering.stackexchange.com/questions/1546...

This is an interesting suggestion to me. Honestly, I consider a lot of other things like type safety and ergonomics of the language before I think about compile speed if at all.

However, that's not to say it's irrelevant. Perhaps I should be thinking about it more. I did notice that they didn't suggest any particular language with this comment. Compile speed seems like a difficult thing to compare between languages in general, since it would be difficult to make a genuine apples to apples comparison.

Pushing implicit logic towards types, keeping non-determinism/impurities at the edges of your program by asserting around edges is a massive win.

The only language that encourage this type of programming (as in it's idiomatic across the ecosystem) whilst still keeping compile times down is OCaml.

Another tip, validation pass. If you have a (complex) data structure like an IR. Create a validation pass, which becomes the specification, of what exactly is legal in this data structure. This is like an assert on steroids.

And.. have a bigger use case at hand. Integration tests are great, but also have a real world case that can be run.

Another way to think about this is that there’s three kind of errors:

- recoverable (return Err)

- unrecoverable (debug assert)

- unrecoverable and tricky (assert)

Until the fast turn around time item came around, and I knew exactly why my current work was bothering me. I gues sometimes you need to be whacked around the ears with a checklist of stuff you should know, before you realize something's off.

assert is supposed to be cheap in production because you remove them there.

In python, this is done python passing the "-oo" flag to the interpreted.

Unfortunately, the community never got the memo:

- if you put it in a lib, nobody is going to use "-oo" in prod, and they will think your lib is very slow

- if you using it in prod, some of your 3rd party deps may very well be using "assert" to make required production sanity checks. So using "-oo" would suddenly break your system.

With large projects you often get modules which have an API boundary, complex internals and clear rules for what correct / incorrect look like. For example, data structures or some complex algorithms. (A-star, or whatever).

Every time I have a system like this, I'm now in the habit of writing 3 pieces of code:

1. A function that checks the internal invariants are true. Eg, in a Vec, the allocated length should be >= the current length. In a sorted tree, if you iterate through the items, they're sorted. And children are always >= the internal nodes (or whatever the rules are for your tree). During development, I wrap my state mutators in check() calls. This means I know instantly if one of my mutating functions has broken something. (This is a godsend for debugging.)

2. A function which randomly exercises the code, in a loop. Eg, if you're writing a hash table, write a function which creates a hash table and randomly inserts and deletes items in a loop for awhile. If you've implemented a search algorithm, generate random data and run searches on it. Most complex algorithms and data structures have simple ways to tell if the return value of a query is correct. So check everything. For example, a sorted tree should contain the same items in the same order as a sorted list. Its just faster. So if you're writing a sorted tree, have your randomizer also maintain a sorted list and then periodically check that the sorted list contains the same items in the same order as your tree. If you're writing A-star, check that an inefficient flood fill search returns the same result. Your randomizer should always be explicitly seeded so when it finds problems you can easily and deterministically reproduce them.

3. A test which calls the randomizer over and over again, and checks all the invariants are correct. When this can run overnight with optimizations enabled, your code is probably ok. There's a bunch of delicate performance balances to strike here - its easy to spend too much CPU time checking your invariants. If you do that, you won't find rare bugs because your test won't run enough times. I often end up with something like this:

    loop (ideally on all cores) {
        generate random seed
        initialize a new Foo
        for i in 0..100 {
            randomly make foo more complicated
            (at first check invariants here)
        }
        (then later move invariants here)
    }

I've been doing this for years and I can't remember a single time I set something like this up and didn't find bugs. I'm constantly humbled by how effective fuzzy bois are.

This sounds complex, but code like this will usually be much smaller and easier to maintain than a thorough unit testing suite.

Here's an example from a rope (complex string) library I maintain. The library lets you insert or delete characters in a string at arbitrary locations. The randomizer loop is here[1]. I make Rope and a String, then in a loop make random changes and then call check() to make sure the contents match and all the internal invariants hold.

[1] https://github.com/josephg/jumprope-rs/blob/ae2a3f3c2bc7fc1f...

When I first ran this test, it found a handful of bugs in my code. I also ran this same code on a few rust rope libraries in cargo. About half of them fail this test.

(That's my main curiosity after reading the post and clicking around. It looks neat but I'm just hearing of it.)

I worked on a 15K loc Python project. We didn't have tests because it was hard to isolate.

We started to puts some asserts here and there on parts of the code we weren't sure about.

After few months, we simply puts assert _everywhere_ and it quickly became a weird but efficient way to cover our backs and it help us to make refactoring fun.

With assert, you can start to think if particular branching is relevant or should be placed on the caller side.

We consider an AssertionError as a developer error and it makes a real difference in investigation time between other errors that could raise.

Encapsulation

Polymorphism''

These were just thrown in, but they both make the code less readable and less performant if they are not really needed.

In my experience while having separation of concerns is critical for software development, the only way to find the right interface for that separation is trial and error.

Iterating on the interface between layers is the most important part of software development.