Mental Model Fallacy (2018)

Once upon a time, a "mental model" was, in particular, a model: some sort of mental representation of a thing, usually simpler and clearer than the thing itself.

I'll give some concrete-ish examples.

- For anyone you know reasonably well, you can probably make a lot of predictions about how they would react to various things. Whatever you use to do that -- which may well be fuzzy and complicated and largely inaccessible to you -- is your mental model of that person.

- Sometimes you might explicitly model someone as a "homo economicus" money-maximizer, who will always do whatever gets them the most money. This is obviously an extreme simplification, but it has the merit of being something you can explicitly reason about.

- Your mental model of the US government might be that the president makes decisions and everyone else does exactly what he says. This would be a hopelessly wrong model, of course.

- If you're good with differential equations, your mental model of a pandemic might be something like the Kermack-McKendrick SIR model, and that might give you useful intuitions for the consequences of (e.g.) implementing social-distancing measures.

- If you're not so good with differential equations, you might adopt a simpler model that just says that the number of infections is an exponential function of time. Even that's enough to make better decisions than many people do.

But here "mental model" seems to be being used to be generalized way beyond that, to include any way of thinking or broadly applicable idea. A few examples from the Farnam Street list linked near the start of the article: "First principles thinking", "Thought experiment", "Second-order thinking", "Inversion", "Occam's Razor". Some of these are useful tools when building mental models. Some of them are useful things to do, or keep in mind, whatever sort of thinking you're doing. None of them is a model.

In fact, almost nothing in the Farnam Street list is a model. But some of the things in the list are useful metaphors, which you could consider to be tiny models, or model-parts. (For instance: "velocity". Velocity is not a mental model, but you might often want to build mental models that include something you could reasonably call "velocity".)

Is the usage of "mental model" to mean "any thinking tool at all" actually widespread? I've seen it before, but I think the other instances I've seen have been closely linked with this one -- all, I think, quoting Charlie Munger. (I don't know whether Munger's own use of the term is as broad as e.g. Farnam Street's.)

I hope it isn't; the narrower notion in which a mental model is actually a model seems to me a valuable one, and it seems easier to find other terms that convey the broader idea (e.g., "thinking tool", "general principle", "idea") than good replacements for the narrower one.

I do think you successfully communicate that you struggle with some of the concepts you're trying to refute.

Just looking at your highlighted statements:

> The most valuable mental models do not survive codification. They cannot be expressed through words alone.

Close to stating a premise but you've gone and blown away the the subject of models. I think you're trying to say farnam street is selling snake oil, but you're now arguing experience can't be taught, which is tangential and generally uninteresting

> When Warren Buffett studies a company, he doesn’t see a checklist of mental models he has to apply.

1) You don't know that and 2) "warren buffet studies a company by running through a checklist of mental models" is a claim you've just come up with to refute (strawmen seem to make up the majority of this post)

> How do you know if a computer program is badly designed? You don’t go through a mental checklist; instead, you feel disgust, coloured by your experience.

No. You've indicated you don't understand design, and again that a model = mental checklist, which is your own assertion

That doesn't negate the value of mental models. Mental models are absolutely required for understanding any complex system. They are literally how we think. They should be refined through direct experience. Listening to experts doesn't hurt either.

What has me worked up is that the examples in the article are just garbage.

A person skilled in tennis or MMA will have no problem communicating to you what it takes to acquire their skill. It so happens the most efficient way to communicate complex physical motions is to model them, then allow the student to attempt them, then critique the student's form. Finally the student will require many hours of practice of the correct forms to build muscle memory. There is nothing in here that says anything about mental models.

"How do you know if a computer program is badly designed? You don’t go through a mental checklist; instead, you feel disgust, coloured by your experience."

You absolutely better have a mental checklist and solid technical reasons for why it's badly designed. Imagine going to your boss, asking to do a complete re-write because you feel disgusted by the code base. How is that going to work out for you? Maybe it's algorithms that are not performant. It has inconsistent abstractions, too much abstraction, or too little. It may not accurately model the problem domain. A feeling of disgust is not going to get you anywhere. Technical knowledge and experience will.

Ok. Clearly this article hit one of my buttons. I think I'm done now.

The problem with it really is as the author suggests the lack of authentic experience on the one hand, but I think more importantly it's that the idea of "mental models" just tries to hand people a bag of disjointed tools.

When you look at what it means to really understand something and you look at say a world class pianist or something, then you'll almost certainly find they have an integrated perspective on what they do. They don't have model A and model B and model C and a bag of fortune cookie wisdoms, they have tacit knowledge and beliefs that are coherent and whole. Really understanding something ironically often leads to the inability to articulate how it is one understands it, because it's just become integrated into how someone operates in general.

Umberto Eco once made the great point that unread books are much more important than read books because known knowledge pales in the face of everything that is unknown, no matter how dedicated one is to reading. And it's the same thing with these mental models. You're not smarter because you know 200 models or 300 models or 400 models, just like reading 50 more books per year isn't going to make anyone any smarter in a sort of simple additive way.

"An important part of becoming a good learner is learning how to push out the frontier of what we can express with words. From this point of view the question about the bicycle is not whether or not one can "tell" someone "in full" how to ride but rather what can be done to improve our ability to communicate with others (and with ourselves in internal dialogues) just enough to make a difference to learning to ride."

Knowledge and wisdom is built on abstractions (this is well established). When i think about how i have distilled the things i know, im certain that its finding the correct abstraction - the correct picture or mental model. for example a mental modal about winner takes all markets allows you to understand that certain markets have this trait and therefore the mental model allows you to identify this class of market efficiently due to a nice terse abstraction. I really don't see the link the author makes between understanding the mental model of a winner takes all market meaning that you should all of a sudden be an expert in how to beat one, you may know the basics but i think most people would also know there is experience, nuance and instinct involved - very indefinite things, unlike a the classification that the mental model portrays. Knowledge is built upon abstractions and therefore mental models are just that, its the classification of knowledge that captures a model (generally static) of the world. In understanding a mental model you get some insight distilled into a neat abstraction. Believing you can win at anything just by understanding a mental model... thats crazy surely !?

This is also why I think a project manager, if a project has one, need to be technically literate to have positive impact to a project's success

It is, in turn, a summary of a 30k word series on 'putting mental models to practice' https://commoncog.com/blog/a-framework-for-putting-mental-mo... — originally published in Farnam Street's Learning Community. To his credit, Shane Parrish of Farnam Street invited me to share my criticisms in his members-only forum, so I reciprocated by putting in the work to back up the ideas in this piece.

Duh. You DO need tools at some point though. You can, in theory, build all of your tools from scratch. Assuming you're already familiar with the different types and paradigms of tools. But it might be better to look at a list of options before deciding which tools you want to deliberately practice using

Your basic Philosophy 101 class teaches interesting mental models like Cartesian doubt. That really shaped the way I thought about things for years to come.

With that said, it’s important to identify models properly. If you listen to MMA fighters or other athletes, you can start to see their incremental approach to increasing training intensity to achieve measurable results. Not parsing that out will leave you with a shadow of a mental model that, in this case, would be mostly comprised of non-core strategies (e.g Always stay positive, have no fear, never accept no for an answer, etc).

Of course that doesn't mean you can learn everything from a mental model. By nature, they abstract away plenty of important details needed for expertise. But they are not deficient in terms of ability to convey some level of understanding.

If you think about it, this has to be the case. In many ways, knowledge is simply a series of mental models built on each other. Yes, much of it comes from empirical observation, but that ends up encoded into mental models.

The other thing about lists is that all the items are roughly equally weighted. In real life, the value is almost never distributed that way — a couple of items absorbed well will often contribute immense value. But we water that down with lists (to make the source sound authoritative), and worse, bury the best stuff down the list for SEO & clicks & “engagement” metrics.

It’s interesting to imagine that as a corollary, this brushes aside almost all punditry, and a lot of context-free college education.

Here’s the thing... a good mental model is worth its weight in gold to a practitioner. There’s something magical about the deep intermingling of theory and practice, with each piggy-backing on the other. To have any shot at that, it is very important to be situated in a context, getting useful feedback from reality. If you think about it, the fields with the best theory also have the best experiments & feedback. Failing that, excessive theorizing is akin to the insanity of a dream world.

I’ll read those books again in a few months/years, after practicing. I’m pretty sure it’ll be like reading entirely new books.

On a more serious note, while we're comparing mental 'mental model' models. My experience has been that it's more useful to think of mental models as a vantage point. A good mental model grants more perspective while bad mental models obscure things. Of course this doesn't absolve you of understanding the basic concepts, perspective won't help you if you don't know what you're looking at.

It's kind of like using analogy as rhetorical device, it works up to a point; but when you get to the details it's not worth extending the analogy.

Mental models can be good for raising questions, but then you have to actually find answers to those questions.

But what is not addressed is that some mental models very clearly can be usefully shared. There is not a clear and distinct line between a "mental model" and communicating fundamental elements of how something works.

For example, my 8 year old was trying to figure out why his walkie-talkies kept making howling, screeching sounds. I explained the concept of feedback to him and he was immediately able to put that knowledge into practice (keep them further apart, or don't have both microphones on at the same time).

I would argue that for tech and business work, communicable models/techniques vastly outnumber noncommunicable models/techniques.

In software development, for example, there is a lot of explicit knowledge to learn, and then there are the mental models about how one should apply that knowledge - e.g. SOLID - but these models do not tell you exactly what to do (in particular, there are all sorts of ways of using the rules to justify bad decisions.) It takes good judgment, honed by experience, to apply the rules effectively.

Is this actually something that people believe? I've always assumed that people reading about doing $THING were procrastinating, rather than doing $THING.

The way I've heard it used in interfaces just means what the user knows (or thinks they know) about how your thing works.

A couple examples from Donald Norman's "The Psychology of Everyday Things" (which was renamed "The Design of Everyday Things" in later editions).

Consider a thermostat, which has a simple dial with an arrow painted on it, with a scale of temperatures printed around it. You can turn the dial to point the arrow at a temperature.

The way this thermostat actually works is that there is a bimetallic strip inside that bends as the temperature changes. When it gets cold enough it bends far enough to close a contact that turns the heater on. When the room warms up enough, the bending of the strip abates, opening the contact, and turning off the heater. Turning the dial modified the distance between the bimetallic strip and the contact. Turning the dial to a lower number moves them farther apart, so the strip has to bend more to close the contact, and so the room has to get colder before the heat turns on.

If you ask people how they think it works, some will know the above. Others will have other explanations.

Some people might think that it is just based on time. The system runs the heater for a variable time controlled by the dial, and then turns it off for a fixed time, and then repeats this cycle. Turning the dial to a lower number reduces that variable time.

There were, if I recall (it's been years since I read it) correctly, a few more explanations given, some quite wrong in the sense that no one would actually build a thermostat that way.

The interesting and important thing from a user interface point of view, though, was that all of them lead to the user doing the right thing when it comes to actually operating the thermostat. If the room is too hot, they move the dial to a lower number. If the room is too cold, they move the dial to a higher number.

The point was that users are going to have some kind of model for how your thing actually works. It is not important that the model they have is actually right--as long as their model leads them to the right control inputs that is fine. Be aware of what kind of models people are going to have, and design your interface to not encourage bad models.

An example of an interface design failure was a refrigerator/freezer Norman had. It had two sliders, labeled "Freezer" and "Fresh Food", both in the fridge compartment. The "Freezer" slider had settings A-E, and the "Fresh Food" slider had 0-9. The instructions said: "Normal settings C and 5", "Colder Fresh Food C and 6-7", "Coldest Fresh Food B and 8-9", "Colder Freezer D and 7-8", "Warmer Fresh Food C and 4-1", and "Off 0".

The labeling of the controls suggest or reinforces a model that leads to someone who wants to adjust the freezer temperature fiddling with just the freezer control, and someone who wants to adjust the fridge temperature fiddling the the other control. Probably something with thermostats in both compartments, each controlling a cooler unit for that compartment, with the sliders each controlling one of the thermostats.

The way that fridge actually worked is that there was a thermostat somewhere, controlling the cooling unit. The output of the cooling unit went through a valve that could direct part of it to the freezer and part of it to the fridge. One of the sliders controlled the thermostat, and one controlled the valve. Nothing really suggested which compartment had the thermostat (assuming that it is even in one of the compartments), or which control was for the thermostat and which was for the valve.

An average user of that fridge who, say, feels the fridge temperature us just right but would like the freezer to be a little warmer is in for a frustrating time of fiddling with the controls. Their model of how it works probably leads to different predictions of control response than the actual model. That way that fridge actually works is far enough away from the models the users are likely to have that the controls should have at the very least been labeled in a way that lets those user know that this fridge is difference.

Maybe label the slider that controls the cooling unit something like "Overall Cooling" and label the valve control something like "More to fridge/less to freezer <--> More to freezer/less to fridge". Still a pain to operate, but at least it is obvious it is going to be a pain. The user who wants a warmer freezer but is happy with the fridge temperature can tell from that labeling that they are going to have to decrease the overall cooling, and change the allocation toward "More to fridge/less to freezer" to keep the fridge temperature the same.

1. Modus Ponens (If A implies B, and also A, then B.)

2. The Hypothetico-deductive model (The guess-and-check scientific method)

And frankly I'm a little suspicious of that second one!

These are better known as "math" and "science," or "deduction" an "induction." And yes, modus ponens is really the only inference rule you need for logic - Hilbert proved that[2].

The other 98% of the algorithms built into our brains are unreliable and cannot be trusted. Consider for example your optical cortex, which is attempts to patch up raw input in a dozen different ways, resulting in dozens of optical illusions, saccadic masking, not being aware of your own blindspot, and so on. We literally can't trust our eyes... or rather, we can't trust the instinctive processing our own brains do on raw visual input. So it is with the other parts of our brain. Or what Kahneman calls "System 1."[3] It's a patchwork of barely functional heuristics.

Scientists learn to shut out that 98% and use only the two reliable systems. Mathematicians take it even further and shut out 99%, leaving only modus ponens and methods of deduction.

People hate that this is true. They want to reason intuitively, naturally. They hope they can patch their hopelessly bugged brains into something useful if they can just memorize and avoid a list of pitfalls. I'm telling you there's a better way. Forget about fallacies. Stop looking for shortcuts like "mental models." Construct rigorous arguments inside of formal deductive systems. Use those to build formal mathematical models that describe reality. Test those models ruthlessly against experimental data, even to destruction.

You know this works. It put a man on the moon, for god sake. It predicted what a black hole would look like, then took a picture of it. It's cured so many diseases and so many problems that our main problem is that we don't have enough problems. Yet people still want to look for shortcuts. I can sympathize with that. We're all busy. But the real choice you face is this: be rigorous, or be wrong a lot.

[1]: https://en.wikipedia.org/wiki/List_of_fallacies

[2]: https://en.wikipedia.org/wiki/Hilbert_system

[3]: https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow