One Way Smart Developers Make Bad Strategic Decisions

> That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.

Past a certain level of complexity, that's no longer true.

_Seeing Like a State_ is a great introduction to this, but I think Carol Sanford's work goes much more into detail. The main thing with the high-modernist view that James Scott was critiquing is that it comes from what Sanford would call the Machine World View. This is where the entire system can be understood by how all of its parts interact. This view breaks down at a certain level of complexity, of which James Scott's book is rife with examples.

Sanford then proposes a worldview she calls the Living Systems World View. Such a system is capable of self-healing and regenerating (such as ecologies, watersheds, communities, polities), and changing on its own. In such a system, you don't affect changes by using direct actions like you do with machines. You use indirect actions.

Kubernetes is a great example. If you're trying to map how everything work together, it can become very complex. I've met smart people who have trouble grasping just how Horizontal Pod Autoscaling works, let alone understand its operational characteristics in live environments. Furthermore, it can be disconcerting to be troubleshooting something and then have the HPA reverse changes you are trying to make ... if you are viewing this through the Machine World View. But viewed through Living Systems World View, it bears many similarities to cultivating a garden. Every living thing is going to grow on its own, and you cannot control for every single variable or conditions.

For similar ideas (which I won't go into detail), there is Christopher Alexander's ideas on Living Architecture. He is a building architect that greatly influenced how people think about Object Oriented Programming (http://www.patternlanguage.com/archive/ieee.html) and Human-Computer Interface design (what the startup world uses to great affect in product design).

Another is the Cynefin framework (https://en.wikipedia.org/wiki/Cynefin_framework). Cynefin identifies different domains -- Simple, Complicated, Complex, and Chaos. Engineers are used to working in the Complicated domain, but when the level of complexity phase-shifts into the Complex domain, the strategies and ways of problem-solving that engineers are used to, will no longer work. This includes clinging to the idea that for any given problem, there is a global solution which will satisfy all local conditions.

>> > A global solution, by necessity, has to ignore local conditions.

>> That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.

So lets rephrase that. A global solution that ignores local conditions will have problems and will likely fail.

Makes sense. In my work I've seen this when trying to get developers on my team using certain patterns, styles, types, conventions, or tools (or the inverse - deprecating them).

Suggestions are usually well grounded (e.g., "let's migrate to this `std` class instead of this old home-rolled wrapper), but sometimes there's some nuance to how something is currently done and deep discussion of the proposal can work through these bits.

> That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.

In theory, yes. But in practice, no (and this is the author's point, I think). In theory, the more "local conditions" you have to account for, the more exponentially complex your "global solution" becomes. (This is the "state" metaphor.) In practice, you can't build that impossibly complex system (and it might not be desirable, anyways!) - so you're likely to try to change local practices in service of a more streamlined global solution. The more you do that, the farther away you move from respecting local conditions.

That depends a lot on the cardinality of the set of Tims.

If there’s one Tim per team, you’ll have 100 Tims proposing different global improvements and 100 teams needing to respond intelligently to those suggestions.

Through enough iteration, all problems can be solved. But, how many iterations will be required to reach a solution that works for everyone? At that point, is there a solid business case for the project?

The scenario reminds me of this story:

https://mwl.io/archives/605

> A global solution can solve global concerns and also allow for local conditions.

Not if standardization is the priority.

I think the key difference is whether the local teams have the choice to opt out or not, and my belief is that they should. If they can, they can solve their own problem if the global solution doesn't work. If the global solution wants to keep them as consumers, they must adapt. If they can't leave, the global team will almost certainly stop responding to their needs over time. Like communism, a global solution is terrific in theory, but human behavior causes it to break down in practice.

Caveat, for small enough problems, good enough solutions, and charismatic enough leaders, global solutions can work. But they all break eventually.

interesting. For long time i've found that negative logic is more powerful/overarching than positive one - #ifndef NOT_THIS is more powerful than #if THIS .. and this article applies that even to agreeing vs not-disagreeing.

It's the large-scale version of taking "DRY" too literally.

Junior devs just repeat themselves because they don't know better.

Middle devs rush into an incomplete abstraction by overzealously not-repeating-themselves.

Senior devs just repeat themselves because they know they don't understand what it would take to abstract out the solution.

Like everything... "It Depends". Don't Repeat Yourself Too Much.

I've worked at bigger companies and there are plenty of folks much higher than 'middle dev' forcing these types of things down the organization's throat.

This kind of consequential decision can happen at high levels. Obviously less often when a truly brilliant developer ends up in a small organization (but that has its own risks.)

In the example, it was determined that they could not afford to let each service solve its individual bottlenecks ad hoc. So a corresponding strategic error was also made/forced at the senior business level.

It's easy to speculate in hindsight, but in this case I could imagine a globally enforced throughput mandate supported by a widely visible and frequently reviewed dashboard, new tools/libraries as needed, and an optional central queue service required to 'compete for business' of the individual service teams.

I can see potential problems with that too, though. In a sense, failure has already happened when growth management is deemed to be too important to be left to capable individuals on decentralized teams.

Enjoyed the article.

I agree.

People use the most practical things at their disposal. If Tim had opted to publish a repository of easy and _simple_ recipes for managing kafka and postgres integrations, while retaining the ability to use original libraries, then I see no reason why it would not have gained traction.

Agree, I too have seen the lack of unification more often then not, because business projects are always local. This client wants feature Y, why build it for all clients right now if only one client wants it, I only want to pay for getting the feature out to the client as cheaply and quickly as possible. And now you've got a single use feature. Then next client comes over, and you can't reuse the feature, so you build it again in a slightly different way by different people, maybe even in a different team, rince and repeat. I see that all the time. And that's just one example of how people get their velocity down to a crawl over time. The only solution then is to hire more and more engineers until you're a huge engineering department maintaining a single product.

Of course, this is such a rampant problem in the software industry that a whole market for reusable standard generic solutions was created. That's why we got the cloud, and the array of SaaS, PaaS, IaaS, etc. And don't forget the entire open source is about standards, being able to reuse existing components and frameworks.

What I think the article doesn't mention is that unifying and creating a standard solution is a harder task then creating custom solutions one after the other for each use case/local context. In practice I've seen people try and fail, but often it's not the person with most experience trying, or the business isn't truly willing to put in the effort to succeed, both of these can sabotage things. And again, because it is hard, you have to be willing to fail the first time, but use those learning to try again, and again, until you crack it. And doing that is often worth it long term, cause when you crack it the efficiency and scale will go through the roof, if your business is smart, you might even realize what you have is more valuable than your current business, and pivot to being a SaaS vendor haha. Or you can keep it secret as a competitive advantage.

> Counter example: Tom standardized a bunch of services... and it worked! Everything is easier and more efficient now.

I’m sorry, but that isn’t really a counter point unless you have some cases to back it up.

In my completely anecdotal experience standardisation never really works. I say this as someone who’s worked on enterprise architecture at the national level in Denmark and has co-written standardisations and principles on how to define things from common building blocks.

The idea was that something like a journal of your health can be defined as a model that can be used by everyone who ever needs to define a journal for health data. And for some cases it works well, it lets thousands of companies define what a “person” is as an example and which parts are the person and which parts are the employee and so on, and it lets them exchange data between systems.

Until it doesn’t. Because all of the sudden an employee is two different things depending on what time of the day it is, because a Nurse has different responsibilities while patients are awake, in some hospitals, and not in others. But because the “standardisation” doesn’t account for this, 50 years of enterprise architecture in the Danish public sector is yet to really pay off.

Some of our best and most successful public sector projects are the ones that didn’t do fanatical standardisation but build things with single responsibilities so that they could easily be chained together to fit a myriad of unique needs.

Now, I’m not against standardisation in any way, but sometimes it just doesn’t make sense and sometimes it does. The issue is that the standardisation approach tends to begin before anyone knows which situation you are actually in.

> How would Tim's system look if they had opted for his approach from the start? How does the 3rd iteration of the system compare to the 1st iteration? Maybe Tim's solution is stepping-stone to something better. Impossible to tell.

Reminds of something a senior developer once told me about rewriting systems: the first iteration is ad-hoc and messy; the second iteration is well thought out but completely over-engineered and the third iteration gets it right because the developers have done extremes and know where the correct middle ground is.

I used to work on the Frostbite team at EA and it was quite the train-wreck. Many game teams spent considerable time moving to Frostbite only to fail and go back to the old game engine, the game teams which managed to move to the frostbite engine were unable to keep up with engine updates and got stuck on old arcane engine versions, the frostbite engine team was split between multiple geolocations and the teams in different locations didn’t get along well and ended up developing silos, and finally there were about a million layers of management in the frostbite team — I heard from old timers that the team used to be much more engineering focused.

Counter example would be all games made using the Unity engine and the Unreal engine no? Or how well RE engine is working out for Capcom. Or how the Decima engine works out for Sony and Kojima, etc.

I have a bunch of friends who work for an EA studio (or who used to work for an EA studio) and they all _hate_ Frostbite. With a passion.

Micro Services need a lot of unified standards to talk to each other.

Maybe everyone understands "unification" differently, but for me making something simpler means making things that are small, independent, and where if they change internally they break nothing externally.

But for this to work, you need common patterns, properties, interfaces, otherwise you can't combine your modules into a greater more complex system.

If you do it right, the system as a whole might be complex, but each piece is simple to understand and reason about and making changes to them is not a risk to breaking the whole system.

But it also means reuse, those small simple independent parts can be reused for more than one thing, that's why the system grows complex, so each micro-service is very much a unified standard on its own, just of a small enough scope to successfully build and maintain for multiple clients to leverage.

But now, if you go to the next level up, you have a problem with the complex set of micro-services you now have, and that complex arrangement gets hard to reason about. That is where people added the idea of Supervisor, to have systems that watch over the subsystems, it serves as a way to unify a set of micro-services into a more reliable and understandable hole.

With a microservices architecture, it's still possible to locally optimize each microservice but then end up with a global architecture that is far from ideal. It seems to me that Tim's starting point was exactly that.

Except when Tim decides to standardize how each micro-service is built, using a custom framework...

And those types of universal standards arrive by a process not unlike evolution - everyone who wants to has a crack at the problem and the solutions compete for a painful decade or two. Eventually a winner emerges from the top few frontrunners, and by 2040 only networking historians remember ALOHAnet or token ring networks.

However, you can't top-down design a universal standard.

With sufficient skill, patience, and wisdom, you may be able to design a standard that's good enough to be widely applicable.

Humans have free will, though (in practice, at least, no matter what you think about the philosophical question).

A standard only becomes universal when everyone chooses to adopt it.

My reading of Seeing Like A State was not "central planning bad". The author explicitly acknowledges that there are many benefits to what they refer to as "high modernist" projects which we all enjoy on a daily basis. My reading was more that large scale projects designed to entirely change the way something is done in order to make it legible to a central authority necessarily throws away an enormous amount of local information and this leads to a tradeoff. You end up being able to build much bigger systems, but those systems are not as flexible.

In the context of microservices that seems like a pertinent point because the purpose of a microservice architecture in the first place is often to allow an organisation to be more flexible and accommodate local knowledge.

> to be a more efficient programmer you either have to learn to type/think faster or you need to build frameworks, write libraries, and codify common practices.

That’s focused on the writing/creating side of the equation. In my side projects, I became a lot more efficient when I decided to put effort into using frameworks, adopting libraries, and copying common practices.

for this specific example, I think the shared library is not the correct approach. Queues work for simple fifo behavior, but in this case they also need fairness (and perhaps other logic, like rate limiting per client, different priority for certain clients etc)

For example, "Customer-A is clogging up the work queue and starving other customers out". The solution to this could look something like linux's completely fair scheduler, where every client is allocated some amount of messages per interval of time. This means messages need to be processed in a different order then they are enqueued, and queues are not good at reordering messages.

I would suggest implementing the queue abstraction as a rest or grpc service, backed by a database like postgres, which holds message payloads and everything related to message state (in progress, retry after times, etc). Now we can implement all the necessary scheduling logic within our queue service.

They hinted at the fact that kafka is not actually a queue, and it especially has problems with fairness if you try to use it as a queue for unequally sized "jobs" and/or unequally sized consumers. Kafka is for the exceptional case; actual message queues are for the default/general case.