Code only says what it does

On the topic of the "theory" (mental model) of a program, I recommend John Ousterhout's book "A Philosophy of Software Design". You get such gems as:

"... the greatest limitation in writing software is our ability to understand the systems we are creating."

"Complexity manifests itself in three general ways... change amplification, cognitive load, and unknown unknowns."

"Complexity is caused by two things: dependencies and obscurity."

"Obscurity occurs when important information is not obvious."

"The goal of modular design is to minimize the dependencies between modules."

This is important to understand when moving through the early stages of a project.

Many projects go through a clear prototype stage (where a lot of disjoint things are written, like a set of utilities to print out information on a file based on the format spec, make files with hardcoded content, etc.), then a system starts coalescing, and finally it's released.

The problem I've encountered is when the prototype is too good. It's an 80% solution, it seems to do everything that's wanted, but the people who wrote it (contractors/consultants/too expensive older devs) aren't the ones who are tasked with finishing the last 20%. The original developers may have understood how to create that last portion with what they'd written, or they may have intended to throw it away [0].

The new developers don't know what's present (and so recreate a lot of existing capabilities), don't understand how to extend it properly (so a lot of copy/paste when the original devs laid out a nice extendable system with generics and or interfaces or whatever the language provides), and the whole thing turns into a mess. This communication between developers is critical, but usually absent.

[0] "This is more of a proof of concept, it does everything you want for converting two file formats between each other, but doesn't scale yet because it's all 1-to-1 mappings, we are working on the intermediate representation now that we have a firmer grasp of what's needed."

"Oh, that's fine, you guys can go work on the next project we've got a crack team that can wrap this up."

"...Ok, thanks for the money."

The crack team never makes that intermediate representation and just creates 1-to-n mappings between each format. The explosion in code size becomes unmaintainable, most of the mappings are the result of copy/paste, and bugs proliferate throughout because, while fixed in one section, they don't realize how many other places that same bug resides in.

EDIT: For the record, [0] started off short enough to be a footnote then grew to be too long for it, and I forgot to edit it properly when I came back from getting a glass of water.

I think I have to disagree with Naur on this, in that people using the Scientific Method don't ship their theories, but we do.

As a scientist who has just succeeded in testing a hypothesis, I now need to go back and document a simplified series of steps that should lead any independent party to the same phenomenon. Once we are on the same page, they can confirm or refute my theory based on their own perspectives on the problem space.

During that process I may discover that I based half of my experiment on another hypothesis that I never tested, or was plain wrong. Now I've discovered my 'load bearing' assumptions. I may discover something even more interesting there, or I may slink away having never told anybody about my mistake.

Essentially, scientists still 'build one to throw away'. We haven't in ages. And my read on Brook's insistence that we build one to throw away is that it was aspirational and not descriptive. And notably, he apparently recants in the 20th anniversary edition (which is itself 25 years old now):

> "This I now perceived to be wrong, not because it is too radical, but because it is too simplistic. The biggest mistake in the 'Build one to throw away' concept is that it implicitly assumes the classical sequential or waterfall model of software construction."

So we are very much at odds with the scientific method. And we have the benefit of hindsight. We have seen the horrors that can occur when you take the word Theory out of context and try to apply it to non-scientific theories. We should learn from the mistakes of others and summarily reject any plan where we do it too.

In other words: next metaphor, please, and with all due haste.

I've had that experience describing the plot of a novel to friends of mine. The novel is pretty complex and covers ideas in crime, internet anonymity, memetics/virality, and then some technical things with vehicles and atmospheric science. Some friends I've talked it through with, they come up with ideas to add stuff and it's like "that makes no sense for this novel, but it's not a dumb idea given how little of the theory of the novel I've communicated to you." Whereas other friends seem to immediately grasp onto the theory and make suggestions that actually fit with the overall concept very well, and usually recommend small changes rather than wholesale plot rearchitectures. It's like an architect coming in and saying "we should move this door two inches so that this door doesn't bang this wall" vs. "We should build this whole house as one story into the side of a cliff."

>...or not care, but all of those things are slow and error-prone

Nah. Not caring is pretty quick and simple. It has served me well!

Seriously I do agree though. One mistake I've seen a lot is assuming that an extensive code base, developed by competent engineers, but which is very complex needs simplifying or rewriting in a simpler way.

Often that complexity is there for a reason, covering platform, customer or situation specific edge cases discovered through hard won experience and feedback from production use.

Twice I've worked at companies where a massive project to replace the core product with a new clean sheet implementation killed the business. That doesn't mean clean sheet implementations are always bad, not at all, but they can like a nice clean beautiful opportunity while actually being blood curdlingly risky.

Thanks for sharing that, I'll read it!

I do agree that comments, documentation, and other artifacts aren't sufficient to solve this problem. The closest I've come is with formal specification, where the intent of the program can be communicated very clearly. Multiple approaches are needed. One of those approaches is continuity: Keeping people around who are familiar with the code base and can pass on this knowledge to others.

> almost nobody ever goes looking for them there

I've seen this claim a number of times and it's always so odd to me. One of my most common activities each day - certainly more common than the activity of writing new code - is reading the commit history for different files. It's always surprising to me to hear that this is an uncommon thing to do.

Edit to add: But I also think comments and documentation of all kinds are good. I don't advocate good commit messages instead of comments, but rather in addition to comments. The more documentation the better.

The only time I place comments is exactly this: to explain why.

Today I just had this example. I placed a little sleep in a loop. But there is absolutely no way to know why it is there. So I inserted a comment to explain the loop is DOSing a server by constantly requesting it and the sleep will reduce the load on that server.

Those comments are not only for others but also for yourself. Even weeks from now it is easy to lose track on why you did things the way you did.

Comments on why are very helpful.

> people rarely make fine grained enough commits to be able to target specific lines or blocks

Take your own advice then: don't hire such people. Why would you insist on maintaining documentation and not maintaining a reasonable change log in your commits, if knowing "why" is important to you?

(I make the advice tongue-in-cheek btw. Better to educate and lift developers up wherever you can than just freeze them out. Not everyone has the experience with these things that we do.)

There's not a one-size-fits-all solution here. Different orgs handle their code history different ways, but having tasted the power Git gives you over certain other VCSes to make the commit log really useful as a record of change, and having taken advantage of that feature several times myself, I wouldn't want to go back.

This follows another comment I just made on the topic: https://news.ycombinator.com/item?id=23743973

> "Clear code shouldn't need comments"

That's like saying articles don't need summary.

A lot of comments can (and should) be replaced with good naming. Sometimes the only reason to introduce a variable or function is to name something. The advantage is that it is harder to not read a name than it is to not read a comment, and it is more obviously a bug when we try to set a variable, or make a function calculate, something not agreeing with its name.

> Let me know what corner cases you thought about when you wrote this.

We actually rarely use comments in code. The best way to reflect the corner cases are tests. It is the comments codified.

> but it can never say why

Usually people don't need to know why on small piece of code unless it's a high-level stuff. For high-level stuff we have architecture decision records (ADR) which records the motivations and situations. But we also keep it very short or people would never read those.

> "Nobody ever updates comments, so they're always out of date" - don't hire such people.

That's a really high standards. It's almost like "We have bugs", "Well, then don't hire people writing bugs"...

There are many ways of breaking comments. For example, someone changed module A, then the properties on module B changed because of A. If it's a test it will reflects on CI, but if it's a comment there's almost no way for people to realize that.

Clear code needs some comments. Not many, though. If you write a numerical algorithm it probably needs an explanation or a reference to where it came from or something like that. Generally, one needs a comment when it is difficult to figure out why something was written that way. In many cases comments are not needed. I perhaps write a comment once every ten files or maybe even less but then it tends to be a rather long comment because the thing explained is very non-obvious. The kind of comment that I hate seeing is when somebody feels the need to tell me what methods are constructors or similar such nonsense.

> "The comments are in the commit messages" - almost nobody ever goes looking for them there

Consider it another tool in the toolbox. I've gone spelunking through git history to decipher the reasoning of something still in use, though comments had been deleted and no one had documented it before me.

Clear code and clear tests shows both the what and the why. With the additional advantage that they can’t diverge and be out of sync like the comments because otherwise the tests will fail. Comments should be used to explain something unexpected. Commenting each line of code is a recipe for disaster.

I've found that questions during a code review make excellent fodder for comments. I.e. if someone has a question (not necessarily an issue) about a piece of code, it's a good bet someone might have a similar question when reading the code later.

yes, comments should be considered code too, but for humans, so later coders (including your later self) can effectively simulate (not but necessarily replicate) the prior code-writing process(es). it's as critical as the code for machines.

which is to say, the comments should be kept concise and current too.

This was a thing at my last job. The lack of comments always made me cringe. While not every line needs a comment, there's a reason why comments would be useful, especially in security software.

I learned a lot of bad habits there and I'm glad I no longer work there.

Their excuses were literally "no one reads comments" "no one keeps comments up to date" "my code is self documenting" etc.

It seems to be fairly common for experienced programmers to point to unit test code as a way to explore or understand an open source software project. I don't doubt that this works for them, but it definitely doesn't work for me. When I'm trying to explore a new software project, the first thing I want to do is find the relevant "entry point," which is arguably the exact opposite end from the unit test code.

Clear code and clear tests absolutely don’t need comments that explain them if they are really clear, at least for me. Comments are extremely useful to explain something unexpected. Commit messages are too limited to explain properly a use case, but linking to a Jira with the proper explanation does the trick. From the tests you can see both the typical use cases and the correct way of using some piece of code and have a guarantee that the code respects the specification. With the comments you have none of this guarantees. I had the opposite experience from you apparently, the people that I worked with that used excessive comments were not up to par with the rest of the team. Also it may related to the mental model, comments just break my flow and make more difficult for me to read the code.

It's a matter of judgement. I've read code in languages I don't know, even, where I could immediately figure out what was going on without reading any comments. I've also patched libraries in ten minutes without reading comments. It's certainly possible to write code that doesn't need comments, though perhaps, we're not the best people to judge when that's the case or not for our own code.

Self-documenting code is not a judgment that you can pass on your own code: that's the role of code review.

95% of the code you write day to day should be clear enough that someone who didn't write it can understand it on first read (the reviewer). If they can't, you probably need to rewrite the code so that it is clearer, not comment it. That's what self-documenting code is.

Instances where the code needs comments to be understandable are rare.

This is quite good actually.

I would add that some elements of design are worth keeping up, like a general architectural overview and the details of some things, like state-machines or specific kinds of statefulness.

It can be done in the comments, at the package level, that way developers can keep it up to date without much fuss.

System tests too!

The last code-base I worked on had do-everything system tests (with unexpectedly good coverage, I'll admit). They were so slow and passed often enough that I didn't immediately spot flakiness.

I got suspicious when the tests started failing regularly when I added new unrelated code.

A good way of avoiding this trap is to require updated comments as a condition of merging a pull request.

I agree that you shouldn't document "how", but when I'm reading unfamiliar code, I find I what I miss is "what", not "why".

To my mind, in well-written code each function should be documenting its contract: what it assumes, what it guarantees if that assumption holds.

(And if it turns out that what you'd write is just the function's name and its parameter and return types with a few grammatical particles added, maybe it's OK to omit the documentation.)

Then if you find that in order to do that you have to write a little essay, or you need terminology that doesn't correspond to a named thing in the codebase, or you're repeating yourself in multiple comments, that tells you something you need to put in higher-level documentation.

Whether or not to document "how" will depend on the target audience.

The code is written for people to read and only incidentally for computers to execute. From that point of view the target audience will matter a lot. I agree that in most cases, you expect the audience to be on level with the code and if you are writing code that is not on level with the audience (for example it is way too advanced) then you are doing something wrong (at the very least not considering who will be working with it).

But not always.

Example: I do with a lot of bright quants. With regards to code that does a lot of complicated calculations, they typically understand "why" but might need help with understanding "how".

Another example: when your audience is mostly junior team you might want to explain "how". My current project starts using more and more reactive constructs and oftentimes I throw in an explanation of "how" when I, for example, get queries to explain during code reviews.

Give me an example, and I will write you code documenting the business context.

Those samples were probably written by someone who had learned how to code in assembly language. In assembly, this kind of comment reminds the reader of the semantic meaning of what is in each CPU register. It could be very useful. Then the person learns C and never drops this habit.

If no one understands what it does, it's not perfectly good code is it? Of course there are rare cases where code cannot be simplified, made more readable or self explanatory and in those cases comments are vital. But the aim should be for the vast majority of code to be easily readable by humans.

Pseudocode, perhaps?

Sure, but then you offload the complexity to the functions used as the declarative building blocks, so you do the documenting in a different place, though you will probably end up documenting complex declarative business logic anyway. (like why is process X that is so similar to process Y require Z different declarative blocks)

Only for simple cases. When it becames complex enough the same problems show.

Example: CSS.

I find functional code much shorter and cleaner, but also when you want to change something along a new axis you need to do a much bigger rewrite than with imperative/procedural/object-oriented code.

OO code: "here's a detailed and long-winded description of what happens that's hard to understand. Ignore 95% of it and change that 1 little detail and hope for the best"

Functional code: "here's a concise and easy to understand description of what happens, understand it fully, throw it away, and create a new, just as clear and concise description of what should happen from now on"

What you're talking about there is a Model Repository. We're building one at the bank I work at, except because our modelling language (or meta model) is based on OMG's MOF we can generate artifacts (code) from our models. You can't do that with TLA+ as far as I know. It's pretty powerful - you can compose models together very easily, as well as generate loads of useful things for data-in-motion.

This fits my theory of programming, and theory of bugs - We take a problem, create a plan, and then write code that implements that plan.

Defects can come from:

   * having/being given the wrong problem

   * right problem, but plan does not actually solve it

   * right plan, but your code did not correctly implement it