The Efficiency Delusion | Hacker News Reader

The Efficiency Delusion

onezero.medium.com

55 pointshunglee224 comments

An interesting aspect of efficiency is that it’s often at odds with resiliency.

For example, consider an airline. Maximum efficiency would involve every airworthy plane being fully occupied making flights that are 100% full. Ok, sounds good. Now, one of your planes hits a sign on takeoff and has to return to the airport and be grounded for repairs. How do you deal with all of these passengers who expect to get to their destination? You can't, your incredible efficiency means you’re fucked when something goes wrong. Now imagine that each of those people is part of some other enterprise that’s 100% efficient. The failure will cascade and before long, society’s only concern for efficiency will be about how to most efficiently crack each other’s heads open and feast on the goo inside.

But, this is just one simplistic definition of efficiency. You could include as part of your model some % slack coefficient, some % of failure for each component in the network, and find some kind of allocation that maximizes expected "efficency". The problem isn't optimization, it's that your model doesn't include enough variables

Can your model ever include enough variables though?

In other words, you can stuff all the variables you want in there, but you will likely still have error - especially with complex systems (regime switches, etc.) - that you did not put in your model. I think economists tried the "model stuffing" technique, and I'm not sure how much it helped in planning something as complex as the economy.

>Can your model ever include enough variables though?

Resiliency isn’t invincibility.

While surely the answer is no for many systems, there are often still be insights to be gained from attempting to model things out as long as you recognize that it will never be perfect.

Your point is indeed interesting, and true in a theoretical sense, but not in a practical sense, I think.

People and companies do build in buffers, because efficiency != best-case efficiency. For example: https://en.wikipedia.org/wiki/Bus_factor If you have a meeting at 11:00, you won't plan to reach the office at 10:59. Personally I plan to reach by 10:30 so that I don't have to spend time worrying about being late, which is a waste of mental capacity. For me that's maximally efficient. So I think your definition of efficiency is wrong, if you get what I'm trying to say.

You’re right that there are different ideas of efficiency. I wouldn’t say the zero-slack version is wrong, but it’s certainly not the only one.

Even with this broader idea of efficiency, the drive for efficiency can cause trouble. How does one define maximally efficient? Do you optimize to handle cases that occur once a month, or once a year, or a decade, or century? Short-term profit makes it hard to optimize for these rarer cases, because you’ll look less efficient for a long time, and the average executive will have moved on before disaster strikes.

Your idea of zero-slack efficiency is interesting theoretically, and I haven't heard it articulated as well as you did. But I don't think it's valid in a practical sense, because you can't just ignore the cost of (say) missing a flight because you were a minute too late. If you ignore one side of the coin, you no longer have a valid model of efficiency.

Going back to your example of a flight that's canceled, the airline should account for it. One simple way is to multiply the cost of that event by its probability. (And multiply it 5x or something, since passengers will be angrier at having an hour of their time wasted than happier at having an hour saved.) When you look at it with this framework, it makes sense why a pilot doesn't skip his checklist because the cost is very high, even if it's a 1 in 1000 case he's trying to prevent. In other words, it's not just how often it occurs, as you suggested.

Another example is how Bill Gates, in the early days of Microsoft, kept a year's worth of salary in the bank at all times. And how just-in-time supply chains broke when the 2008 financial crisis hit. Another way I heard this stated is that the rich are not the people with the most money, but with the most options. Or, hypothetically, a company with 100 engineers who builds iOS and Android native apps dedicating one engineer to building a hybrid app in Flutter because wasting 1% of engineering is time is better than betting the entire company in a wrong direction, thus wasting 100 engineers' time, though the "plan B" engineer's work is unlikely to yield a launchable app.

You also rightly point out issues like short-termism and misaligned incentives, but those have been discussed many times already, so I'm not going there.

Maybe it would be better to say, a naive idea of efficiency without accounting for exceptional conditions can leave you worse off overall, and it’s easy to forget about them when focusing on the day-to-day.

I would keep in mind that efficiency/optimization is only possible _given complete knowledge of the system_. You can only truly optimize if you know all the inputs.

For any uncertainty you have not modeled, _you must have slack in the system_. This is true of your personal workflow, your professional environment, your romantic life, your business. If you aim only for optimization, you are implicitly optimizing _for the things you have modeled_ - this can often come at the cost of the things you have not.

I would say that, perhaps loosely, optimization is the delusion of overconfidence. Slightly less optimization, and more redundancy, is good caution.

More info here: https://pbs.twimg.com/media/DdvogC9VMAEz9Y1?format=jpg&name=...

That, like elsewhere, is a naive view of efficiency - in the same way trying to apply mathematical logic to reality is a naive approach. Just like reality is best modeled in probabilistic terms, optimization can be done on probabilistic models. Redundancy comes out of it naturally, if you put costs and uncertainties in your models.

Is it a delusion? I don't think so. I've seen very real efficiency improvements thanks to tech and software. For example, do you know how frictionless it is to travel these days compared to 10-20 years ago? You can book a flight, reserve a table at a highly rated restaurant, get turn-by-turn directions, convert currency, look up foreign language translations, book a hotel or AirBNB, order delivery, take photographs and share them with friends and fam instantly, etc, etc. Much more efficient than the old way. You'd risk getting lost, disconnected, lose time to search, lose money to excessive travel expenses, etc.

And that's just vacation travel. The author casually mentions that Uber and Lyft supplanted "hailing cabs" as if it were not much of an innovation, when we all know what the experience was like if you were in any city less dense than New York...it was a pain in the ass to get a cab and it was usually way more expensive than it was worth and the cars were hardly ever clean and you had no idea what kind of driver you were getting. You were also lucky if they took a credit card.

The author isn't saying efficiency doesn't exist. The thesis is "We believe that if technology can make some aspect of our lives more efficient, we’ll get back free time to do the things we actually find meaningful."

Of course things are more efficient now. But the real question is-- does that make our lives better? Is an efficient vacation a better vacation? What would an "efficient" vacation look like (the idea sound gross to me. Getting lost and overwhelmed are part of the fun for me).

Simply put, efficiency taken to a logical extreme is bad and we should acknowledge that.

On the individual level: If your perspective on the world is one of efficiency, you will never be able to eat the fruits of your increase of it. You will just pack more things into less time, less space and less money. However if you are like e.g. my favourite sysadmin, you will be incredible efficient in order to have more time for the fun parts. If he weren’t efficient be wouldn’t be able to foxus on stuff he likes because be would constantly be out to extinguish fires. Because he is efficient, he must not.

Efficiency is about your optimization goals, a guy like him is optimizing for free time that is why he optimized the rest of it to such a degree, that he can afford it without changing the whole thing for the worse.

He could easily also have used that free time to do more work, but he refused.

My stance may be a little stronger than the author's here, but it's something I wish I understood a decade ago.

I used to think that more free time was a key to happiness. Turns out I don't believe that at all now, and most of my free time I spend browsing internet garbage.

I realized that I enjoy challenges and being needed is actually very fulfilling to me, and not something I can just artificially make happen in my free time. As such, perhaps in a more efficient world (where I don't help my neighbor fix their car, don't cook with roommates) I am less happy.

This isn't to say efficiency isn't often good, but I think some of the things it may cost (e.g. fewer possible friendships) may not be worth the price.

One thing I've noticed since moving to a city where driving is unnecessary, is how good it feels to drive occasionally. While I think a big part of that is the physical sensation, I also think doing these types of "menial tasks" can put your mind into a mode that is not possible when being entertained by a smartphone/kindle/whatever.

Currently on my 35 minute commute, I can be using a smartphone about 90% of that time. Does that sound efficient? Sure. But I think the article hits upon a real point, if we're going to "save time" on a particular activity, it had better be a worse activity than what we end up replacing it with.

> Take the myth that once self-driving cars spare people from the burden of needing to drive to work and pay attention to the road, they’ll be able to focus their attention on invigorating and rewarding activities, like reading for pleasure during the commute. More likely, employers will expect the workday to begin the second you enter the vehicle. Instead of being disburdened, more productivity will be ratcheted out of us.

It does free us up to do more reading. Downside: that reading will be approximately 100 percent more email.

> What really resonated is that some of the engineers you interviewed described experiencing inefficient situations as really troubling, like smelling something awful or tasting something gross.

Watching some people do their daily work so inefficiently makes me squirm - I look away.

However I've seen that drive for efficiency in a wide range of professions: it definitely isn't peculiar to engineers (e.g. the shortcuts a chef learns).

I also explicitly deoptimise some things so I can enjoy them more e.g. varying routes.

I think what the theme for.me and my urge to optimize is, is that I like to automate/optimize processes that I hate doing repeatedly, while processes that I like doing are allowed to be inefficient (e.g. i make.my filter coffee by hand using a hand grinder).

If you optimize things on a personal level you yourself are responsible for what to do with the newly won spare time.

On a society level this is different however, because not working means no job and no job means no money. If they make the job you did more efficient by optimizing and automating all the bits you liked and leave you with all the bits you hated (and a lower wage), you will have a hard time cheering over the optimization.

There are good points here. On the other hand, the world's wealth is built on efficiency improvements and in some cases they can be literally lifesaving.

I guess it's just about knowing when to turn it off.

This article is spot on. Engineers increasingly define the course of our daily lives, and they've been taught, almost universally, that efficiency is not only a value in and of itself, it is the highest value we can pursue. But we're all destined for the grave, and if we truly want maximum efficiency, it follows that we should promptly kill ourselves. Clearly efficiency cannot be our only guide, and I think the current technocracy has gone much too far down that dead-end.

So is efficiency the new bias? Efficiency is also closely related to excellency in terms of evaluations. And we don't teach these things in school. I've noticed in practice people use cover-all expressions like "you will know it when it's right", or "oh, that would be too complex". The required level of explanation seems to escape us.

Though I think suicide is a bad example of "efficiency," I think the larger point stands.

I think it's a good exercise to think of what a highly-efficient world might look like and whether it would be pleasant at all (one language, perfectly regular architecture, increasing isolation, more interactions with machines instead of people, less chance/wonder in life)

Reminds me of Universal Paperclips.

Reply on news.ycombinator.com