Simple Systems Have Less Downtime (2020)
gkogan.co
gkogan.co
There are problem domains where there is simply a minimal complexity required implicitly. You can’t write an insanely simple program that will render vector fonts, simply because the problem inherently has to have a fixed amount of complexity. Anything more (accidental complexity) is of course bad, and “clean” code or whatever should try to minimize its amount, but it is harmful in my view if we keep believing that it could be made simpler. One really bad example is this list: http://harmful.cat-v.org/software/
Their primary challenge is making money, and that part probably is better as a web service, but it makes it worse from technical point of view. Just like with hypothetical FaaS - worse technically, but you can have adds.
The sentiment behind that list is okay but it does demonstrate a lot of ignorance The biggest joke is simple tables or file systems as a SQL alternative. If you do that for anything that needs more than just a map you will eventually end up with a badly implemented buggy slow relational database.
I feel like I am turning into a bot for posting the Out of the Tar Pit paper:
http://curtclifton.net/papers/MoseleyMarks06a.pdf
This changed my understanding of computer science and our product virtually overnight. We are using a hybrid model of Functional Relational Programming (see section 9 in the paper). This is in production right now and its clearly the right answer for managing complexity in our product.
If you think "eww i have to learn some FRP language" - No. You just need to tack SQLite onto whatever preferred language you use today and model all your business logic as SQL queries over properly-normalized tables. If you can achieve this with 6NF, you have an infinitely-extensible domain model. If you don't know where to start, 3NF is the safest place. Most humans tend to think in terms of 3NF when referring to the business entities.
Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns.
Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query.
I'm not saying don't normalize. But normalizing creates its own problems too you may need to think about.
If it were up to me (and it almost never is, because I'm not a DB expert), I would ALWAYS normalize as much as possible when designing a database. I really can't even wrap my brain around why large (and old) databases ever get into this state. But I've seen it at two different employers and it's very painful (and costly) to deal with. My take was that these databases were probably originally set up by people who had no idea what they were doing, and ended up locking the company into a shitty implementation that kept the company crippled 15 years later. But that's just me.
> Let's say you have billions of rows of event data you with to perform summary counts for by a few different key columns.
Guess what? 99% of people on here don't have billions of rows of event data they need to regularly aggregate. Problem solved.
Such a terrible tradition in our industry in focusing on outlier cases, or on what FB/Google/... might need when making technical decisions.
All sorts of different patterns and architectures have to be brought together to make sense of it.
But the technique described above still has its place within such a system. For example - you boil billions of data points down to a handful (several thousand) facts, such as events that occurred or anomalies. Then those several thousand facts can be analysed standalone (as described above) without going back to the source. Etc.
having built a real time analytics solution a few years ago because we wanted one we controlled for our startup, it doesn't take much to get to billions of rows of event data in the modern web.
Analytics on large-ish data is definitely a common challenge. And sure, CRUD is even more common. But knowing what your options are for each challenge makes sense.
While I'm not necessarily arguing in favour of using Sqlite for this, it most certainly can do this up front during ingestion.
So yes, sometimes you do end up sacrificing normalisation, but often that means keeping a normalised form, and having code that can re-generate summary data from the normalised form.
Absolutely agree. That is why we use in-memory SQLite projections of business state that are scoped per-user-session so that row counts never exceed double digits in any given table.
For us, adding indexes to these special database instances would actually make things go slower.
> You just need to tack SQLite onto whatever preferred language you use today and model all your business logic as SQL queries over properly-normalized tables.
was to think "well that can't work for _all_ my business logic". But I'd like to see how this idea works in practice before I jump to that conclusion.
My first thought was the same as yours (no way that works for everything).
It seems like a neat idea. But I can't imagine that anyone has ever done this for anything non-trivial. I want some sort of compelling argument that this works at all before I'm willing to accept this as anything else than a pipe dream.
For us, the more complex the business gets, the more justified this path becomes. Our business fails to scale if we have to write custom code for every customer. Writing custom SQL per customer (against a common schema) is far more tenable.
Are you thinking of some specific logical determination that would be infeasible in SQL vs in code? Or, is this more about webscale arguments?
Finally, I'm not going to venture in (let alone actually live in it myself) without someone sitting me down and carefully explaining the structural integrity principles (and other logistics) that make the cardboard skyscraper possible.
Hearing business rules in sql works for some unknown entity doesn't really give any information about the viability for anything else. Maybe the initial modeling is really hard, but all your customers have near identical concerns, so you only pay that cost once. Maybe having multiple customers is actually the key because if one customer's needs violates some assumption that makes the whole thing work, you can tell them go to someplace else. Maybe your customers only have five business logic rules apiece that are all well spec'ed.
What's the domain? The industry? What problems are actually being solved? Can we have an example (even a contrived one)? Is there an open source example? How about a mathematical proof or even principle that argues for why this can be expected to always (or even nearly always) work? How about a youtube video with someone describing the approach?
EDIT: How about this. Show me a partial parser using this approach (grammar rules can be considered business logic, right). Just parse function declarations in C or something. I'll be able to project such an example onto what the full solution would look like to see if I believe it could actually work in general.
I'm disappointed to see there wasn't a response because I was pretty willing to entertain the notion of business-logic-as-normalized-tables. But I needed, like, any evidence.
The translation of that into requirements is by itself one of the largest sources of accidental complexity.
And even the value statement sometimes is wrong and leads to unnecessary complexity too.
Good point!
If a system is hard to interface with, the complexity is still accidental, it is just outside your own system and maybe out of your control.
Even if you’re a “middleware” company that connects multiple shitty systems together, you’re still adding zero value to anything but your own pocket. The complexity is still there.
We still have the right to call a spade a spade. In the past I had to say “no” to managers about interfacing with shitty systems. Sometimes you CAN make a difference.
The trope I see all the time is enterprise tooling in small projects. Small teams basically moon-lighting as devops 50% of their time and reducing their capacity to solve the actual domain problems significantly.
The domain complexity is something you can't remove, but complexity you've chosen to introduce through tooling should be hard thought about, and the less time you spend thinking about stuff outside the problem you're actually trying to solve the better.
YAGNI is so hard for people to really grok.
Get the revenue model right and suddenly every feature is cheap; get it wrong and every decision is infinitely expensive.
Yeah. This is both true and pointless.
Nobody codes things too complex on-purpose, at least not normal people. So it's not the difference in types of complexity, it's the difference in our ability to understand accidental and essential complexity.
I find that I do a really poor job at this, and I'm the first to stand on the soapbox and rail against systems being too complex. Just like programmers naturally introduce bugs into code without realizing it, programmers naturally introduce complexity into code without realizing it.
We may theoretically be able to talk about the differences in type, or how to manage each, but our real problem begins in our conceptual models inside our brains or various types of problems, and this happens a long time before any code is ever written.
In my personal practice I've come up with a few gimmicks that help me refactor my preconceptions. It is very difficult, however, for people to adopt practices that continue to inform them that they make lots of mistakes. Everybody wants to code as if they're conquering the boss-level at the end of a game. We sell coding, tooling, frameworks, and practices to other coders under the assumption that they're going to have a blast, not that they're going to continuously be reminded that they're screwing things up.
So yes, I agree, but without moving past that statement into something more tractionable, it's more of a truism than a starting point.
This resulted in them coding things too complex unintentionally, so while not on-purpose, it was absolutely a useful conversation to have. It's a good starting point.
I don't have a good single rule for a "second step," though. It's going to depend on the details of your project, by and large, though I think some principles to try to strive for include composition, encapsulation of details, and so on - nothing new or startling, but just adding the "is this essential or accidental" lens to the decision-making process.
However, if you bring up the topic, then take a look at some code or architecture? Then suddenly we're talking about all the ways we might have done it better/less-complex if things had been different.
Most every coder I know understands the topic, and most are even willing to go on at length about how important it is, including me! (grin). It's the actual application where things fall apart.
>> I don't have a good single rule for a "second step," though.
I do. It was bugging me so I spent a couple of years coming up with one. Seems to work great for me. YMMV. I do not believe it is as context-dependent as most in our industry seem to think. (It's very much problem dependent, though. It's just vastly more related to the business problem than the technical considerations. [Discussion goes here about tech-related problems not related to the solution foisted on the solutions team])
It's not so simple — I've seen people create over-architected monsters because they thought everyone coded like some consultant's magazine article/talk/book claimed, because they were trying to be taken seriously and didn't think they could push back, because they had talked their way into a job they weren't qualified for and tossing around buzzwords was a way to obscure the fact that they had no idea what they were doing, because the client insisted that all real projects must run in the way the consultants in expensive suits assured them everyone serious runs them (ignoring massive differences in scope and resources or long term success), or because the institutional incentives made it easier to say their project would handle everything anyone had ever thought might be useful into one application rather than working with the actual users to build multiple smaller, simpler tools.
One way to think about that is that it's neither accidental nor essential complexity but environmental. Fixing that can be quite hard but it's often work the political capital if you have it because the alternatives might not be survivable.
If simplicity is always the right solution, then why do so many programming languages have dedicated parsing libraries for .csv, which has to be the world’s most simplest data format ever?
The basic rule for CSV is: Don’t. Or at least use a library which emits RFC 4180-compatible results. If you need to parse some co-called “CSV” non-RFC-compatible monstrosity, do whatever you need to parse it, but don’t have any illusions of it being in any way “standard”.
This is acutely summarized by Larry Wall's "Waterbed Theory of Complexity" which is basically Einstein's "Make things simple as they have to be but not any simpler than that". i.e if complexity is like water -- not very compressible. If you compress that complexity (a part of the bed) -- it will manifest elsewhere in the bed as more awkward/expanded because it's all connected.
Many discussions on complexity do make this distinction.
To me it’s sort of implied that it’s unneeded complexity when we talk about it as something that should not be there and it’s making our life hard.
That's indeed a pretty bad example, regrettably of a website often shared in programming circles. The author simply listed his beliefs about stuff he doesn't like vs stuff he likes, and never bothered to justify them. In many places the list is obsolete, too.
Also, UTF-32 seems way simpler conceptually than UTF-8, so I don't understand how UTF-8 ended up "less harmful" and UTF-32 ended up "harmful".
Same sort of issue makes people use microservices not because of their true advantages but simply to enforce boundaries between features because so many people lack discipline to make a well-structured (distributed) monolith.
This is coupled with organizational dynamics where to get resources you have to prove value. Proving value is a lot harder than scaling/rewriting a software system, because if you need to scale a software system, you have rewards and money attached which are good business drivers. If you need to prove business value, no one cares (and you won't get resources) until you do.
It's a fine line to walk for sure.
Also true with personal finance:
* https://canadiancouchpotato.com/2016/01/25/why-simple-is-sti...
Well said Dijkstra. Some of the my favorite algorithms are beautifully simple, mostly in the way of being naturally recursive. My compiler course in university involved a lot of recursion, but everything just flowed. Sometimes I'd mindlessly write some code and just assume I'd recurse down the AST, and everything just worked.
Obviously simplicity isn't always achievable in large software, but hopefully there are many simple pieces involved that come together in a clean and concise way.
"That's been one of my mantras - focus and simplicity. Simple can be harder than complex: you have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains."
I've posted this here before, though got no comments. But yes, firefighter/arsonists get more promotions and kudos than careful thinkers who don't bother setting fires to fight.
EDIT:
One past discussion here (2015): https://news.ycombinator.com/item?id=8940820
The refactor was incredibly difficult and launched without issue and I am probably back on track for a promotion, but I have started to make noise about the incentives being askew
Choosing the hard path again I see.
1. Can't finish what they started
2. Are going to rewrite stuff unnecessarily
3. Are willing to throw their coworkers under the bus
So it's not like the main developer handed something off that was unfinished. They handed something off that was finished but wrong. And because of that we had to replace the engine while in flight, so to speak.
The team isn't able to get promoted because they're busy doing boring, undervalued work. The engineers responsible for the mess ultimately end up ahead.
If your goal is to get promoted then clearly the best path is to take shortcuts and not act in the long-term interest of the team. You'll get promoted for it. If your goal is to build a product that won't be burdened in operations/maintenance then you have to move slowly, correctly, and you'll be setting yourself back.
This was my experience at a FANG.
Far easier to jump ship at opportune moments for fast tracking a career than fighting dysfunctional internal incentive structures.
What makes things worse is the ability to stitch together twenty different AWS services together because - well - you can.
Solving hard problems in simple and maintainable ways, I would argue, IS the job. But what I see is engineers solving problems they don't even have, for THIS IS THE WAY. Also, there is a FAANG guy on the team that claims that's how they did it over there.
K8S could only have come from Google. It's a reflection of their corporate structure. Conway's Law and all that. Much like microservices. It's a reflection of their internal chaos. And yet every company that isn't Google tries to emulate them and fails. Then they wonder why microservices suck so much. It's not that microservices suck. It's that Google sucks. Google sucks so much that microservices are pretty much the only way they can operate as an entity.
This is a real problem for system architects/security professionals that need to maintain a bird's eye view of the system - a job that gets increasingly complex with every additional microservice once you hit critical mass.
This is not to say that K8S does not work - they do, but for certain problems. The cargo-cult mentality, however, threatens to introduce complexity in systems where there is need for none.
I'm very happy building simple and easy to maintain systems without flashy features. I couldn't care less about a promotion.
Complexity can result from features which pay for themselves. Complexity is not automatically always wrong.
At scale, complexity solves problems that you might not otherwise realize exist. Some of these problems are intrinsically complex, they can't be solved with a simpler solution. Not without "simple" organically growing into a much worse mess than the dominant complex solution that was designed by acknowledging complexity from the start.
But what I want to respond to is your trying to explain that "KISS is always the right approach". Rules of thumb like these are cheap.
They cause people to apply the same patterns regardless of the situation. Dispensing folk wisdom indiscriminately only serves to stop all thought and analysis.
Zealous rule-based engineering is the deathrattle of good systems design, and it makes me ill at ease.
Meaning to say, I find KISS almost universally applicable, there are not many situations where you wouldn’t want to try and simplify (if you can).
What I wrote, and then said several times was that:
"Simple systems fail in boring ways. Complex systems fail in fascinating ways."
I gave examples from Chernobyl, Air France Flight 447, etc. as examples of such. It's a topic that fascinates me and I had hoped to give several more such talks - but alas, the pandemic.
In my opinion, the best discussion of this topic is the book _Normal Accidents_ by Charles Perrow[2].
[1] https://www.colorado.edu/libraries/2019/11/08/cyberinfrastru...
Some of my observations on complexity were touched upon in this interview, however:
https://console.dev/interviews/rsync-john-kozubik/
Specifically:
"rsync.net has no firewalls and no routers. In each location we connect to our IP provider with a dumb, unmanaged switch.
This might seem odd, but consider: if an rsync.net storage array is a FreeBSD system running only OpenSSH, what would the firewall be ? It would be another FreeBSD system with only port 22 open. That would introduce more failure modes, fragility and complexity without gaining any security."
... and I recommend these console.dev interviews in general - they're usually very interesting.
There's no glory in simplicity.
Add something - you
Remove something - you + other owners + all other users
Which explains why system migrations only succeed when championed by a VP+.
Lawyers and bureaucrats careers are based on this complexity of criminal justice systems and government administration, respectively.
He spends about 5 minutes talking about how removing things is the most essential part of a good design: https://youtu.be/t705r8ICkRw?t=833
Good test suites alleviate fear and allow people to make changes (alterations, additions, deletions) without fear, greatly increasing the speed of development and delivery.
At my last job, we hired a new team to transition our stack to a new architecture. They built it, and SOME components of the legacy stack DID get replaced and deprecated. Some did not. Which is why across our two stacks, we used no less that 4 totally different, separate database implementations. 3 different NoSQL cache implementations. 2 different types of kubernetes clusters, (RKE and EKS). 3 totally separate monitoring and alerting systems. (just a few examples, but the whole problem is way too big to list out here). The worst problem was that staff who designed the legacy systems had either left, or didn't remember how they built it or why they made all the choices they did. So the ONLY effective means of knowing whether a legacy component could be replaced, was to simply shut it down and wait to see if anybody complained. (we called it a "scream test"). In some cases, customers would simply stop paying, and nobody (but finance) would notice for MONTHS. We'd still have people logging in from non-paying customers. Or in other cases, the customers had totally shut down their own operations, so there was nobody to pay, and nobody to call us and say "hey, we're not doing business with you anymore". Engineering never wrote a formal way of validating the whole system, so when features would break, the only way we'd know is if a customer noticed (assuming they're still using the feature), and called Support.
Now, this only happened in the 2 years I had been there. In the previous 15 years, I imagine this had happened several times (aborted modernizations/migrations).
In all that time, nobody had been allowed to sit and plan out how the whole system could be simplified, so that the whole organization could even understand what they were deploying and paying AWS for.
Often, this leads to homegrown solutions that are never at feature-parity with COTS options, but have the benefit of costing more.
If they buy something instead, they festoon it with a bunch of customization and external, parallel tools that must be kept in sync at the data and configuration level for everything to "work".
What they NEVER do is re-examine their assumptions about how they must work, or interrogate why they can't go the same route as the bulk of the market on these points and thus have a simpler, more maintainable, less frustrating system in place.
It's not quite this bad, but it's ALMOST like a company insisting that "well, here, for historical reasons, 2 and 2 are 5, not 4, so we have to do . . . . "
It's exhausting, but OTOH, well, to the extend that my employer is involved, we tell them our recommendations but ultimately bill by the hour.
It is not simplicity that makes for less downtime, it is unnecessary complication that does the opposite.
I spend time to complicate my applications a little bit to make sure there is no downtime, something pretty important when one of the largest banks on Earth will stop along with your application.
The simplest solutions would typically not be able to ensure no downtime operation. I need code so that I can do rolling upgrades and I need code so that my application can partition work and rebalance it reliably as cluster map changes.
The problem starts when you start overdoing it. Maybe you are expecting too much in terms of guarantees. Or you want a simple guarantee but then you duct tape to it a huge and complicated clustering solution. Now you have a lot of problems. Your team doesn't know how it works. Your team doesn't know how it fails. It is not easy to tell if you are obviously integrated the right way. And so on.
The goal, as usually, should be to "keep it simple, but not simpler than is necessary".
For example, the approach we have chosen was to get by with as little guarantees as possible implemented as simply as possible.
We decided on immutable data. We decided objects being saved as documents each with entire state of the object after each change. This costs a lot in space and processing needs, but you know what? It is fine. I work for a bank after all. The one think that costs more than space and processing power is downtime, and that's what we are trying to focus on. We know how to deal with duplicate data.
So another rule of thumb: try to find compromises in your application, use them to replace hard problems with easier problems.
Making an application super reliable is hard problem. Adding more storage space and memory is (relatively) easy one. If you can solve hard problem by replacing it with an easier one, you are winning.
Over-engineering does not mean "making the thing worse". It does if your engineers don't know what they're doing. If they do then the added complexity increases reliability.
What I think is important most of the time instead of zero downtime is predictable behaviour when failures occur, so that your system doesn't end up in an undefined state where you don't know if you can recover without data loss.
Mobile networks? Power delivery (basically all utilities)? Broadband internet? Factories? Payment systems? Air traffic control? Any internet services at all?
One can think that only Google or Facebook need to maintain high availability, but basically any internet services needs to do that or face possibility of loosing clients.
Smaller company may have less clients but the downtime still affects their clients and in turn them the same way. Loosing 10% clients after snafu is as painful when you have 1 million clients as when you have 100 clients.
It is also not about having absolutely no downtime -- this usually cannot be guaranteed. It is about having less downtime.
Networks certainly fail all the time, and power delivery issues aren't uncommon either; the downtime just tends to be localized and if you depend on your internet connection or power, you have backup links and UPS systems that reduce or avoid the impact of downtime.
I never said you shouldn't strive for high availability, especially if you're moving vast amounts of customer traffic, I specifically made the argument against "no downtime", because people often seem to think that if your system doesn't have five nines of uptime it's unsuitable for handling "real" traffic, and that to achieve enough high availability you somehow need a highly complex system.
Power outages of a few minutes to a few hours are utterly normal. Power outages up to a few days due to summer heat and winter storms are part of the rhythm of life, depending on where you live. Facilities that really care about power continuity have batteries and generators (although these aren't perfect either, we once lost a datacenter to a transfer switch maintenance).
Broadband is notoriously flaky, to the point that cable technicians' vague arrival windows are a meme. Serious businesses get several independent connections. Even consumers can now fall back to tethering their phones.
Credit card authorization gets skipped during downtime. Actual payment settlement occurs in nightly batches, which humans have many hours to shepherd and patch. FedWire keeps banker's hours. Stock markets suspend trading when necessary.
Stopping the line is a normal part of the lifecycle of a manufacturing process: when something goes wrong, when there's going to be an upgrade, even for regular scheduled maintenance.
Most internet services have some downtime.
Tracing that trough Google's Ngram Viewer, I found the extended form, "complexity is the enemy of reliability" in a short item in a 1959 issue of The Economist (Jan 18, 1958, volume 186 https://books.google.com/books?id=aDsiAQAAMAAJ&q=%22complexi...).
That in turn was referencing the UK's Annual Report of the Chief Inspector of Factories for the Year 1956, which had been published in late 1958. Among the factors assessed was factory and equipment reliability, which scaled precisely inversely with the number of components in equipment.
https://www.worldcat.org/title/annual-report-of-hm-chief-ins...
Software isn't quite the same as mechanical equipment where individual components may wear, but each software component does exist and interact in relation to others. And eventually that complexity comes home to bite.
A full definition of "simplicity", as with "complexity", lies somewhat in the eye of the beholder. But extraneous components are a problem.
(I cannot find either reference online, presently, though I've copies of both documents. I'd tracked the Economist issue through Google Books, and received a copy of the report through a friend. I'm told that the UK generally has excellent availability of government publications, though none of https://www.gov.uk, the National Archives, nor the British Library seem to turn up the Factories report presently.)
Perhaps "somewhat", but not entirely or arguably even mostly:
The greatest act of simplification I did was reduce the problem from a continuous one, some one wanted to filter on distance and I moved it to a set of three buckets.
This discounts all the things argued out of existence or replaced with a sql query.
Which reminds me, the longer you can keep something relational the larger your chance at reducing complexity, because you can project down to a lower dimensional space to solve specific problems. 80% of work is in choosing that right lower space and making the projection. Once there, all problems are flat.
Are fitness functions universal?
(I think I may have swapped the ranges on "Threshold sensitivity", which seem more reasonably to be "low/high" than "high/low".)
Since this is from 1956, its copyrights have already expired (or waived, as the case may be, see https://lists.wikimedia.org/pipermail/wikipedia-l/2005-May/0...), so scanning it is a fair game, but if the British government (probably the National Archives) scanned the document, it would add another 25 years* to the digital version (unless released under OGL).
* At least in Britain, in the US simple photographic copies are not considered as derivative works and therefore not in copyright.
I turned up a copy through a friendly online librarian (as with the Economist article). Given the age of both publications and their rather minimal commercial value, the hoop-jumping is annoying.
That said, the general availability (if not legality) of information online has absolutely exploded over the past decade or so.
Abesence of copyright in published works in the US is a specific and legislated exception to exclusive rights in copyright, and applies only to the US Federal government. Not states, counties, cities, or other governmental units. And not foreign governments either. 17 USC 105(a)
https://erlang.org/download/armstrong_thesis_2003.pdf
As well as from the way the Erlang language and runtime are organized.
Just as a ship with dozens of people got replaced by 13: the complexity was encapsulated in simpler interfaces.
Diesel motors: just change this lever. Up is fast, down is slow. But it took a century to refine this absurdly complex device until this simple interface became possible.
I think the point here is: don’t build your own diesel motors. It’s unlikely you’ll beat a good model and your interface to it will probably be worst. Remember: people take years learning from their own mistakes to hide the complexity under simpler interfaces layers.
Unless your company IS building diesel motors.
Hidden complexity in the case of the diesel motor was made possible also because of know failure modes (how many years until the low hanging fruits all got solved, how many decades until the not so easy got too?) and standardization of its components (how many centuries just to understand and standardize the universal joint https://en.wikipedia.org/wiki/Universal_joint?).
More complex than humans, with wives, kids, need for downtime, hazard pay, insurance requirements, mental and physical limitations, and so on? A lot of the complexity of humans is invisible to us because we're just used to it.
What happens is that it starts simple and becomes complex as more features are added often with limitations attached e.g. time, money etc.
So yes this guy migrated from Marketo to Hubspot and it was simple. But the idea it will simply remain that way is laughable.
As such, the obvious answer is “it depends”, and making the right trade-off is rarely properly captured in a single rule of thumb such as “always choose simple”.
A genuine question that I wonder about: How does a single server protect you against someone in your DC pulling a plug by mistake or your cloud provider being unreliable? If it’s a database it seems like a huge potential for going out of business.
I guess if you are running single-server databases you better have a good backup strategy anyway, but is that a risk people and businesses just accept?
The tradeoffs involved in avoiding some of the simple "the cleaner unplugged the DB server" introduce complexities which may themselves cause outages which are much harder to solve.
If instead of using this single server (with manual fail-over in case it would fail) you'll start to create complex HA cluster from multiple servers not having enough people to properly design, test and maintain it you can end up having a less reliable system.
That’s why you have redundant power supplies with separate power cables.
The recent fire in a French DC saw serious disruption across all kinds of industries in many European countries. I know a sales engineer in a hardware manufacturing place, who was back to pen and paper until the IT provider figured out restores. But it wasn't that bad, as both the clients and competitors were decimated too.
I've heard this from many engineers over the years.
But what I've often found is that they were never involved in the decision making process and so aren't aware of all of the business requirements. Once they are made aware usually they agree with the choice.
Common example being the business requirement to have high availability.
It is even the simplest things that are ignored such as:
- what happens to your db query when you have 1 million records?
- what happens when the cache goes away?
- do you even use a cache?
- why are you treating your cache as a DB?
- can we run 2 copies of your service?
- can your service scale up and down?
- does your service have state?
I've never understood the hype about CQRS. To me it looks like a nightmare for a tradeoff that you "may" only need at some point in time. Particularly given GDPR, HIPAA, PCI and other compliance frameworks that require you "the right to forget". All books about CQRS talk and talk about the wonders of it and when it comes to that detail they just limit themselves to say "yeah, you should consider it"... but the reality is that the implications are huge.
We frequently run into customers who want to implement some feature or system, which really should just be handled by their load balancer, database system or some standard Unix tool.
I currently work with a customer who have an... interesting approach to api-gateways. We already ripped out 75%, because it either does NOTHING or have been replaced by rules in ha-proxy, which we use anyway. The last bit of code can just be integrated directly into the single remaining service behind the gateway.
The same client is running Redis, which seems innocent enough. Fairly simple, easy to manage, but it only hold ONE item. A 150kb JSON document... That's can just go in the memory of the signle process which needs it and which is responsible for refreshing the document anyway. Worst case is that it takes 60 seconds more to restart the service.
Just add K8s to the mix and you are starting from a relatively complex system.
Is this really a thing? I have yet to see a single case of it.
OTOH what I found more often than I could count is the belief that complex systems or tools are necessary because the problem is complex (it isn't) and the company's situation and requirements are so special (they aren't) -- not intentionally but simply due to lack of time and lack of contemplation of the problems and available solutions.
But what I've seen multiple times: someone is enthusiastic about a new technology and advocates for using it in a project at $job even when a simple, but equally or better suitable for the task option is available.
It may sound offensive, but my impression is that many developers are like children - they like to play with new toys and quickly bored by old. And in software industry they have an opportunity to choose new toys and be paid for playing with them.
Also I've seen many discussions on social networks where people talk that nowadays it is hard to find a good job if you don't have X, Y, Z on your CV so they may feel pressured to try these new tools even if they don't enthusiastic about them.
The setup :
- Distributed file system using GlusterFS
- DNS load balancing using Amazon Route 53
- PostgreSQL HA clusters using Patroni
- A WireGuard mesh topology between all instances.
Even though it was much fun setting this all up, the ballooning complexity of all this outpaced the benefits and it felt more fragile than where I came from. I decided to just scale vertically and keep it simple.
Most of my refactoring these days is coming up with more simple, more manageable solutions.
That’s the root of quite a bit of evil in software. A closely related motive is a desire to show off.
Play is good for learning but not for production systems. You’ll regret it later when you are up at 4am on a Sunday morning troubleshooting some Byzantine stack.
The antidote is the realization that simplicity is harder than complexity. Simple but highly effective systems are the ones that should inspire awe and admiration. Complexity is a sign of an immature design or a lack of high level conceptual thinking. A system should be only complex enough to capture essential complexity (problem domain requirements) and no more.
One more thing… there is a ton of submarine marketing in our industry that encourages complexity because it’s profitable for vendors and cloud providers.
The design you outline leads to more cloud resource consumption, more lock in to cloud platforms, and eventually a need for lots of service mesh, config management, and orchestration products that if the project grows will eventually start costing money.
Yes but its also driven by keeping oneself employable.
" I built a simple system to do X" is not a winning conversation in an interview. GP's complex system is what ppl say they built.
You can also point out how fast you built it, since simpler systems often take less time.
A simple system such as a hand-written web server is very likely to crash. It's very likely cannot serve many users if we don't make it serve requests concurrently. What's worse, one tiny exception in one request would bring the whole system down. It stops serving any request to any user.
A relatively complex system with a self-recovery mechanism added goes a long way compared to the former. The Erlang implementation is much more complex than "an Erlang without processes". Kubernetes is complex, and it definitely gives us less downtime than simple scripts we have written before we use Kubernetes.
The real world is chaotic. If a system needs to strive for less downtime, it needs to have some features that mimic biological creatures, instead of being idealistic simple.
> Kubernetes is complex, and it definitely gives us less downtime than simple scripts we have written before we use Kubernetes
There is a point in between 100LoC hand-written web server and K8s cluster. E. g. you need to serve static files and load is small to moderate (say <=1Gbps) you can do this in two ways:
1. Create a K8s cluster where HTTP traffic will be dynamically proxied to a container running on some set of nodes reading files from GlusterFS cluster.
2. Install nginx on a pair of physical servers (with disk configuration depending on the load: from HDD mirror to SSD RAID 10) and something like CARP for HA IPs shared between them.
2nd option is much simple and in most cases will be more reliable: in theory K8s option can have higher availability but it requires much more effort to not screw up something and much harder to troubleshoot when something goes wrong.
Sure, it doesn't do it for millions of people, but it's drastically less failure-prone, despite the lack of resilient design.
What boggles the mind is people are often sold the cloud even though their org isn't serving millions of users, and could just as easily operate as a box in a closet.
Capex vs opex may play a part here. Also, companies in general seems to have transitioned to using services in general. 20/30 years ago companies had cleaners in them. Now everyone uses a cleaning service.
Either corporate accounting is some mystical art that makes money appear where there isn't if certain practices are followed, or there's some collective mass delusion that opex is just better? I don't know, it's a concept that truly baffles me.
Why big established companies prefer high OPEX to lower CAPEX is less clear.
Take, for example, website hosting. You can put your full page on a single server, but then you're prone to the machine going down, the disk having errors etc.. A more complex system, like Kubernetes, will be far harder to troubleshoot, but it will not go down just because one machine died.
Or, as an engineering example, take planes: They're a lot more complex than ships. Fixing them when something is wrong can take long. But that complexity is needed so what when something fails, you get to the airport instead of going through a rapid unscheduled disassembly.
Also, even with fly by wire there’s sometimes (or often, or always, no idea) a backup, eg a direct electric / hydraulic link.
But simple is not easy. In my experience, it often comes after complexity.
My general approach to simplification is to first get it working; even in a complicated fashion, then start removing stuff, until it stops working. If I can't get it working on the latest simplicity, I am forced to add the last thing back.
But I can usually get it going, which may sometimes require a rearchitecting.
One of the happiest times for me, when I'm writing code, is when I get to toss out a whole bunch of painstaking work that I did.
I'm going through that now. I'm working on an app that has necessary complexity, and I'm finding ways to toss out code all over the place.
- Joe Armstrong
It does leave one wonderful if the niftiness is worth the pain, or if there is a way of getting the good bits without the complexity.
By default Money does not know or care about conscientious design. It wants functionality and it wants it now, future be damned. This is not irrational. Usually there is no future.
The problem start with your unlikely success, and yes now you have money to throw at it, but ironically the people that are best suited to wrestle complexity to the ground are not super motivated by money. They are motivated by a peculiar kind of beauty that is, by hypothesis, missing in the successful, complex system, and so they will not want to contribute, money or no.
There's legitimately nothing "simple" about the massive container ship system described in the OP. It's a complicated semi-autonomous cyber-physical system with multiple complex failure modes possible in the software, electrical, hydraulic, and/or mechanical domains. The reason said system has both the superficial appearance of simplicity to its operators and the high-probability of low-downtime is because its very complex systems and subsystems were fastidiously designed and built using tools, processes, and practices which helped manage and understand all that complexity. Not because it is "simple".
Self-hosted or cloud-hosted Kubernetes is a highly complex system. But using it to power your internal tooling (e.g. Jira, Confluence, GitLab, Jenkins, whatever) over classic, manually-installed-on-VMs deployment has the advantage that you can drop in anyone with experience in Kubernetes to manage your workloads.
Core company secrets belong on premises of the company owning the secret, not in a cloud where it is fair game for security services and criminals of all kinds and nations.
That’s very hard to implement
I absolutely agree with the premise - but some of the anecdotes are laughable.
We can and will debate what programs are "simpler" all day, it's all subjective... except for one metric: line count. Line count is objective.
Take pause before rejecting any change that uses fewer lines of code.
It takes an awfully good abstraction to beat simply having less code.
Then there's the subjectiveness of what you actually consider a "line".
Still, some argue that an 8 line for-loop is better than a 1 line map or something, but a map is more constrained in what it can do than a free style for-loop.
And 8 vs 1 lines in some corner of the program is less important than whether or not we make it a coding standard to write and use AbstractBeanThingyamabobs for all our stuff, etc.
printf("Hello, World!\n");
Put("Hello, World!\n");
There are also analysis methods (names escaping me, and Google fu is weak today) that look at loops, procedure calls, dependency graphs, and other things to attempt to discern (should be treated as guidelines and not rules) the complexity of a program using objective metrics.
Cyclomatic complexity https://en.wikipedia.org/wiki/Cyclomatic_complexity might be what you're thinking about.
Hit the nail on the head. We can't fight thermodynamics, we can only cheat by drawing lines and pumping entropy one way. "Simplicity" is obtained by hiding complexity away. Modern container ships are reliably run by 13 people because of a huge system of builders and maintainers for everything from diesel engines to navigation systems.
If people know a language, like regexes (or APL as an extreme case), they can hide a lot of complexity.
In short: smart is stupid.
― Mark Twain
It might not be as efficient or convenient, but that wasn't the question. If the efficiency of convenience of two competing system is sufficient, the less complex one will have fewer parts to fail, and cost less downtime.
So for example for a "who touched this cabinet last" sheet, pen and paper would indeed often be better than smartphones.