Simple Systems Have Less Downtime

Don't solve problems you don't have.

I agree wholeheartedly that simple systems have less downtime.

I would like to add a line from the talk that I give:

Simple systems fail in boring ways. Complex systems fail in fascinating, unexpected ways.

rsync.net storage arrays typically have multi-hundred day uptimes. But across our entire network, our actual aggregate uptime is unimpressive - perhaps 99.99% ? Our SLA dictates 99.95.

But when they do fail they fail in very, very boring ways. There are usually zero decisions to make in response to a failure.

> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[9]

* https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law

I don't have experience with Looker, but the unseen complexity of no-code tools often leads to very complex systems with interactions that are hard to understand. A dashboard may not have that many interactions with other tools, but the black box nature of these kinds of tools usually leads to significant downtimes when there's a small detail not working as expected, or where you need a small customization that wasn't foreseen by the no-code API creators.

No, the containership is not a simple system. It's very sophisticated one. It takes massive engineering effort (literally historical effort and knowledge) to build, massive resources for the material and outsourcing, very complex (like the pic) infrastructure in case of repair, satellites to guide the ship and check the weather, operators from all around the world to track the containers, massive ports to load-unload the shipments, etc...

The operation of guiding the containership through the sea is a simple one but only once you have everything of the mentioned above in order.

If you are building software (a full SaaS solution), you are not in the business of guiding the containership. You are in the business of building either the containership alone; or the whole related infrastructure (or parts of it). That's a huge undertaking that requires massive engineering efforts. It's not a simple task.

If your task is a simple team that guide the container, you might make money for a while until everyone else figures out that you have no edge there and they can do your job for much less.

The Instagram/Tinder examples are a bad one. It ignores the fact that to find the right (addictive/highly viral) app, you'll need (again) massive psychological knowledge and expertise into how primates brains work. We don't have that, so engineers are brute-forcing by making too many apps until one hits the jackpot.

Your system should be testable, debuggable and it should allow a human to step in and take control if needed. An open system is generally better than a closed one.

Any line of code, especially one that introduces a new concept, like a class or a method, is in “addition” to the root problem we’re trying to solve.

The closer the code/data/org-processes/whatever matches the fundamental problems we’re trying to solve, the less complexity we introduce.

However, you cannot “remove” complexity! A common mistake I see in software companies is using a single Jira ticket for one or more reports, one or more conclusions, and one or more action items.

For example, if we get reports A, B and C, they may all go into a single Jira. Oh, but C is a totally different thing! It’s not a “duplicate”! Ok now someone needs to extract the info from a comment into a new Jira (which no one will... and it’d be a mess if they tried).

Now a bunch of discussions are happening in the comments of this mega-Jira. 3 conclusions come out of these discussions — perhaps what the cause is and what should be done about it.

Then let’s say there’s two action items, one is a temp fix and one is a long term fix. Unfortunately, I usually see people is the same Jira for both tasks (“Assign this back to me when the temp fix is in so I can do the real fix”).

But splitting Jiras up is frustrating. The actual splitting is frustrating to set up, and it’s a pain to browse them.

So most Jira workflows remove fundamental complexity (reports vs discussions/conclusions vs tasks) and introduce extraneous complexity (tonnes of fields and workflows that don’t necessarily apply).

One should “embrace” the fundamental complexity of the problem(s) they’re dealing with.

This applies to programming languages, too. Python is easy but when you do advanced stuff, often without realizing, you start to grok many complex details to make it work or make it work in a performant way. On the other hand, Go (and Rust supposedly, but I don't have experience) is simple. It doesn't let you do many things and that results in fewer ways of doing a specific task but when you do it, you feel safer because in the end, it does look and feel simple although you spent 1+ hour than you'd normally spend compared to coding in Python (this assumes you're doing some advanced stuff, not quick start).

"Choose tools that are simple to operate over those that promise the most features."

Doing this very much depends on your requirements, doesn't it? What good is a simple-to-operate tool if it doesn't do what you need it to do? Sure, maybe you can simplify your requirements; then again, maybe not.

The example of troubleshooting a whitepaper form very much depends on the design of the system. Maybe there's a reason for having multiple forms. If they all shared common architecture and the problem lies in what they share, it's not necessarily more difficult to troubleshoot many as opposed to just one.

Moving from Marketo to HubSpot is good and well - for now. What about years from now? How did they end up with such complexity with the Marketo solution? Both solutions and requirements evolve. Down the line, you could end up with the same difficulties with HubSpot. I think it depends in part on how the organisation handles change.

Lastly, I agree with the sentiment of Mateusz Górski, who commented on the page: is there any hard proof beyond anecdote to back up the article? If you expand the article's concept of a system to beyond just software, all you need is a hardware fault in a third-party hosting company with lousy support for your simple piece of software to be down for weeks.

One non-software system was a rather expensive UPS/generator. It was meant to trip on a power loss and provide X minutes of stable power. In reality, it was incredibly sensitive to minute power fluctuations and would trip and then immediately fail, dropping all supplied power in an instant. The system was dramatically more reliable with this unit simply disabled.

Due to my lack of web development knowledge and massive time crunch, I built it with no framework, zero best practices and the most basic code I could build. The code is atrocious, and would not scale at all but it does what it is supposed to do and works because I did everything in the most basic way, there are no gotchas. Straight html, basic javascript and php on the backend. It interacts with google docs as well. Its almost to dumb to break.

Sometimes we really do complicate things with all of our new fancy frameworks, microservices, states, etc.

With that said, it would be impossible for anyone else to maintain and if anyone posted the code on the web I would probably be unemployed. But it works, very difficult to insert new features though.

I'm not sure if the simpler solution - hubspot existed when the company started hacking solutions together.

Also, the move to hobspot was relatively easy because the business processes and workflows were already well known and defined.

Starting from scratch, fighting the fires as they appear, a bit of patchwork seems unavoidable.

Everything is obvious in hindsight.

I guess the real lesson here:

Simplify once in a while. Like Facebook just did with messenger.

There's an ELK stack + plugins for logs and alerting and saltstack to trigger some basic automated sysadmin tasks.

I left 3 years ago and the entire infrastructure has run on autopilot with zero downtime. Content editors and web developers still add to the content constantly. There is some dynamic content on the site handled by JavaScript and a few light APIs that I built to support the site. There's over 5000 pages of content. This simple infrastructure powers a website with a low-5 digit alexa rank for a business bringing in about a billion dollars a year.

The entire infrastructure bill comes in under $100/mo and they no longer need to employ any backend engineers. We negotiated a contract agreement for me to support the site if they need it. They haven't.

* Yeah, but often they do less / are less effective during uptime.

* "simple" is an inspecific concept. Is the implementation simple? The API? The hardware? The requirements?

* "simple" is a vague concept. Is a 1000-line C program more complex than a one-liner script in a fancy higher-level scripting language? Yes, if you ignore the 1,000,000 line abstract virtual machine, JIT compiler, interpreter, large standard library and what-not. Otherwise - maybe so, maybe no.

> Modifications before additions

I actually agree, but:

* Managers don't like it.

* Sometime this downtime happens before there's ever been any uptime...

* Tomorrow you need something else, and the changes for that require re-engineering the parts you already re-engineered to best accommodate the previous addition.

Need to handle machine failures? Push a bugfix without causing downtime? Maintain a single state across the redundant machines? Handle load spikes? Ones created on purpose to DOS you?

All of this requires additional mechanisms, which can themselves cause failures.

A lot of Software Engineers / Developers in my experience tend to over engineer projects or they are mandated they do so by so their dev leads. It is easy to get caught up into abstracting almost everything away almost for the sake of it while it provides no real benefit.

When the shit hits the fan, there's an 80% chance that your most senior members will find the problem. They may find it quicker in the simple system, even, and others might get there first. But that last 20% is a very long tail, and even longer in a complex system.

In a complex system, many of your members cannot participate in the triage process, because they don't know enough to know what's relevant. They slow down the people trying to work the problem trying to learn new things (good) or offering low-probability scenarios (bad).

There is no 'All Hands on Deck' scenario for the complex system. To be responsive, you have to kick some or even a bunch of people out of the room, and once they leave they can't really participate.

The first phase of a triage is getting a tight repro case. Many avenues are blocked until that happens. And some people have a knack for repro cases but aren't so great at debugging. With more people you get a higher quality repro, which cuts a lot of time off the rest of the process.

Part of debugging is the cost of the verification versus the likelihood. Simple Systems afford the opportunity for people to test out unlikely but plausible scenarios that are in the long tail, without distracting from the more 'boring' checks already being done.

And when bugs are identified prior to deployment, a simple system means you can hand the repro case to the responsible party and expect/demand that they get their fix on the first try.

https://mcfunley.com/choose-boring-technology

> When new requirements come up, the tendency is to add layers on top of the existing system—by way of additional steps or integrations. Instead, see if the system’s core can be modified to meet the new requirements.

I like the mention of this as it is - at least to me - really counter-intuitive.

- Thomas Paine, Common Sense, 1776

This is a well written article, but the metaphor is a little stretched.

Marketo and Salesforce are the "ship builders". Marketers are the crew operating the ship.

The declarative tools in enterprise apps provide Marketers with conditional branch and execution capabilities without code structure, so the average Marketer just keeps adding layers of complexity. Whereas a Developer would continually refactor and make old code obsolete.

The crew operating actual ships do not have anywhere near this level of access to the ships configuration, hence more reliability in the system.

Such a good line. I've seen companies killed by hard-to-follow arguments advocating for technical complexity which serves feature goals which serve business goals.

'Good strategy is simple' from some strategy book is sometimes right; arguments that hold business goals hostage to technical complexity are trouble.

> In the end, the system I put in place had 97% fewer processes (from 629 to 20) while providing all the same capabilities. A bug that was found a few days later got resolved in four minutes.

It‘s called refactoring. It‘s a constant process, that none likes to allocate time to, because it does not move metric immediately. Especially in marketing with a very fast changing landscape you go from 20 to 600+ very fast.

What are alternatives to commonly used complex systems, like Docker/Kubernetes, Active Directory, Remote Access or just shared folder structures? How to keep the functionality of centralized management while making it simple? Would be happy to hear your thoughts on this.

But after a certain point, it's no longer enough to make it simpler. You have to get a lot more involved again, and this time the complexity has to be intelligently focused around the end goal (be it performance or uptime).

Good luck attracting good developers with that. We use something over engineered but we have fun. The customers don't care but we do.

5 pages of pure awesomeness

is the complexity appropriate for the goal (i.e. Everything should be made as simple as possible, but no simpler. )

Hence, often a required step to fix complexity is to change the goal, in such a way that it can be implemented with less complexity.

I am suspicious of this statement. Is the author now supporting the company and defacto replacing the guy that left?

It is simple and it just works for its intended purpose.

I am working hard to apply this to the design of Webase [1] which is a #nocode platform that is inherently complex. But I really believe that if we get the design right it will support a large number of sophisticated use-cases.

Thank you for sharing!

[1] https://www.webase.com